Slashdot Mirror
Ask Slashdot: How Best To Disconnect Remote Network Access?
An anonymous reader writes "Is there a device to automatically disconnect network or otherwise time limit a physical connection to a network? The why? We are dealing with a production outage of large industrial equipment. The cause? The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days. Obviously the main issue is that they were able to do this at all, but reality is that IT gets overridden by the Process Control department in a manufacturing business. They were warned about this and told it was a horrible idea to allow remote access all the time. They were warned many times to leave the equipment disconnected from remote access except when they were actively working with the supplier. Either they forgot to disconnect it or they ignored our warnings. The question is, is there a device that will physically disconnect a network connection after a set time? Yes, we could use a Christmas tree light timer hooked up to a switch or something like that but I want something more elegant. Something with two network jacks on it that disconnects the port after a set time, or even something IT would have to login to and enable the connection and set a disconnect timer would be better than nothing. As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks and repeatedly make blunders like connecting process control systems directly to the internet, use stock passwords for everything, don't install antivirus on windows based control computers, etc. How do others deal with controlling remote access to industrial systems?"
Cant think of anything else...
Tomorrow is another day...
Enable port security which ties each port to a mac address of the other device connected to it so that all ports on the network switch are locked down to just the devices white-listed to connect. Write down what port your gear is connected to which you want to limit access to the internet, and then simply disable or enable that port to allow it to connect.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
iptables lets you specify times if you're using a linux box as the firewall, otherwise consult the fine manual that came with your equipment or consult a professional with said equipment. This is bog standard.
fucking kidding me.
Time is an illusion. Lunchtime doubly so. ~ Douglas Adams
There are some firewall/access devices/content filters that restrict access on both time schedules and destination. Maybe talk to your network administrator?
boom goes the dynamite....
Obviously, just put the device behind a firewall. If the firewall operates in bridge mode, it won't use NAT, so the people who insist that their equipment is directly connected to the Internet won't know that it isn't.
The real "Libtards" are the Libertarians!
Part of this depends on how they have remote access...is it dial-in? Are they connecting to a jump host via IP connectivity? Is it a VPN? The solution depends on which of those they use, because it's all different. You can use a relay to open/close the actual circuit to the phone line if they dial in; I know a few power companies that use this as a safeguard for their power substations that have dial-up access. If it's a jump host or VPN, then the details of that solution define the approach.
But here's a question for you...what about having a limited time to have remote access would have kept this from happening? From what it sounds like, the process control people would have let them in anyways. And then...what happens if they run out of time, halfway through whatever they're doing? Or even more interestingly, what if they screw everything up (again) but then blame it on being disconnected while they were in the midst of doing something, so they can put the blame on you? This sounds more like a people problem than a technology problem.
For your security, this post has been encrypted with ROT-13, twice.
You could stick an old machine with 2 NICs running Linux in the middle running a simple proxy written in nodeJS. Since you've written the proxy you can write all the rules you want. With node it would be incredibly easy.
I'm a DCS system admin at an oil refinery. We keep the DCS and business lans totally separated, and that directive is driven from the top down. If anyone asks for remote access we just let them know that's NOT going to happen - end of story! It can be a pain getting files from one network to the other (patches, etc.) but certainly worth the effort.
If this system is using an Ethernet connection, just get a Linux or *BSD box running with bridged Ethernet interfaces or pay for a decent smart switch. Heck, you could probably do it in Windows -- that supports bridged interfaces too.
Simply disable the interface connected to the device you want to protect whenever you don't want outside access. With a Linux/*BSD box, this could be accomplished with simple scripts. You'd probably have to write up a simple manual procedure to do it with a switch or Windows box.
Assuming you have managed switches a simple crontab entry pointing to a shell script can open a connection to the switch an admin down the port that its plugged into. If you want to get really fancy you can have the outbound traffic going via a transparent squid proxy / iptables so you can tell when the port is in use, and keep logs of the connection state.
You can also go with a non-NAT firewall (bridge mode), which will block incoming connections while the device / people on the inside wont know anything is there.
Honestly a timer on an unmanaged switch isn't a bad solution, it takes any technical skill out of the equation, its (assuming the timer doesn't fail) hack proof, and does not require and maintenance / patching to keep secure.
> As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks
If you're going to call someone inept, you better make sure you're not, especially if its your own FUCKING FIELD.
How on Earth do YOU get to make fun of other employees at that company? I can think of at least a couple of filtering methods more elegant than a freaking christmas tree timer and I'm not even in IT. If all departments' staff quality is the same as IT I just hope that the "large industrial equipment" is not something that can affect other people.
Filtering access on a per-request basis is one thing, and I see how that's critical and can't think why you haven't implemented this already. Filtering access on a per-timer basis is the WORST WORST WORST idea ever. If I could make that any more caps locked I would. There are SO many things that can go wrong with a blind timer-based disconnection that I won't even bother to list them all, I will just paint the simplest of pictures in a newspaper title: "Incomplete update to a CNC machine leads to hands being sawn off".
Do yourself a favor and change jobs.
How does the OP have a job?
The real IT guy was fired and replaced with a mid level sales manager. Anonymous submitter for a reason. And maybe they sell Christmas tree light timers and happened to have a few lying around.
“He’s not deformed, he’s just drunk!”
I think the OP is missing something.
I do process control. It's not manufacturing, but that part is irrelevant anyways. The issue at hand is that process control has shifted to control systems that are networked. There are options that don't use ethernet/ethernetIP, but they're increasingly going the way of the Dodo.
We're in a strange time when control systems are increasingly being networked, and the guys that used to do control/automation (and used to do it with relay/hydraulic/pneumatic) don't have the necessary training to integrate the systems correctly. Most IT people don't understand how control systems work and the implications of changing network configurations.
The way forward is to merge IT and process control. Unfortunately, that's easier said than done.
check your contracts before doing any thing you may be on the hook for the full cost of that large industrial equipment after you break the contract
...is a post incident review with support people involved, and their management teams, along with directors and executive involvement to identify what the problem was that caused the business to be inoperative for the duration of the incident, what policies and procedures need to be followed going forwards, and so on. Once policies are established, solutions that support those policies can be implemented.
As an example for your situation, since a vendor was involved in an upgrade, that should have been part of a scheduled change. The change should be documented ahead of time as to what is being done, what systems are going to be touched, and who the responsible parties both within the company and external to the company are for that change. Included in the documentation should be the fallback plan for dealing with issues that crop up during and after the change, within an appropriate test window that is included in the change window, as well as clearly defined backout procedures. "fix and fall forward" or equivalent statements are not, and should not be, considered acceptable plans. Wherever possible you want to have documentation attached that the procedures involved have been tested in a suitable test environment. (This may not be possible in situations where a test environment would cost as much to prepare as the production environment.)
As far as limiting remote access, as others have pointed out, such limits are trivial based on what type of remote access is in place, and what policies are established. At the very least account authorizations required for performing changes on production devices should require someone in house approve that authentication, be specific to the time when those changes are scheduled to happen, and should not allow similar access to devices or types of devices not involved in the change.
You never know...
The best solution is to use this event as a jumping point into securing it right... No matter what technical solution you come up with, the weakest link are the people. Education, some firings, and getting a better vendor are the real next step. Remote access can be a marvellous tool to getting problems straightened out without flying people in, but it sounds like these are the kind of people you wouldn't let walk unescorted in the plant...
More Caffeine. NOW
The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days.
This isn't a technology problem.
Through their incompetence, they caused damages. Collect your evidence, hire a lawyer, and make demands. If they refuse to pay, sue them.
Watch how fast they start caring about doing remote upgrades more carefully, competently, and with customer involvement. The only thing companies collectively care about is making money. At the very least, you'll cause their liability insurance rates to go up.
Please help metamoderate.
All Internet connections must cross a Firewall. Disable inbound connections, done.
Seriously, this is a question?
One of the more amusing hardware hacks that I've seen in the physical security industry is when a customer hooked up the power lead of the remote access device (modem in this case, could have been a switch or something else) to a key card reader. The security staff would badge the reader, the output would turn on for 1 hour, and then shut off. The really nice thing about this is that now they could track who enabled the remote access. If the vendor wanted to connect from 3:00 to 6:00 for example they could create a time zone that would turn the output on for that time period, and only certain people had permissions to configure time zones. Worked pretty well.
Our salescritter had jokingly told this same customer, "This system can do everything except make your coffee in the morning." The customer took that as a challenge, and the next time we were there we found that he had set the system up so that when he badged in the front door for the first time in the morning it would fire a relay that would turn the coffee pot in his office on.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
A Christmas tree light timer ??? How does the OP have a job?
You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...
I just got done with a contract at a large bank (It's one of the 50 largest companies in the United States)... all their deployments are run off USB drives hung off servers at their retail locations, they have 512kbit backhauls to their corporate locations, run DHCP over the WAN, have no QoS, and I kid you not -- about 5% of the managed switches have been forced to 10mbit half-duplex.
And since they're so security conscious, all the workstations have drives that are encrypted, have antivirus that runs every 4 hours, whether you're using the system or not, a couple other "intrusion detection" apps that also run, sometimes on overlapping schedules, sometimes when trying to patch the operating system... and for the bonus round: An account used for software installation that has full local admin to every workstation... and has a password that's the same as the account name.
-_- Attaching one of those appliance timers to a switch to shut it off at predefined intervals seems so stupidly obvious, but when you realize how stupid the average person is, and then realize that the ones stupider than that work in HR and Accounting, you quickly conclude the same thing the rest of us in this industry have:
Just drink your damn beer and try to drown out the stupid. Thinking about it will only depress you. Trying to do something about it will get you fired. Trust me... there is no faster way to get fired in IT than doing your job well... because you'll get noticed by all the incompetent asshats that HR and Accounting let in, and they'll form an alliance against you to get rid of you. And for the super jaded special bonus round... trying to get shit done will make you realize that the reason you can't get anything done is because everybody has silo'd themselves away with crucial documentation, settings, or knowledge, to assure themselves of continued employment. Start poking around, and they'll feel threatened. When they feel threatened, they'll find some way to go behind your back and make you look bad. Do this enough times and management will consider you an agitator and... ker-chop.
If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.
#fuckbeta #iamslashdot #dicemustdie
That's fine, until the process control guys unplug from your nice managed port, run a cable across the floor and plug into a port that you're not actively managing. And they will do that. If you don't think so then you haven't worked in that type of environment.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
And then require the supplier to be on site to do the upgrades to make sure that they do it right. Screw anyone that complains, bring it to the highest level of the organization with hard numbers of how much a stop will cost.
Total isolation of mission critical networks is the only thing that works.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
This isn't my field, but I think you should do nothing. IT's job is to provide network access. Process Control's job is to keep the machinery running, and if they fail to do so despite your warnings, it's their ass on the line.
Yes, "not my problem" is a classic way to make a workplace awful, but consider this: if Process Control can't get a software update to their machinery because you've blocked it, and something bad happens (worst-case scenario, a machine kills someone), then it's *your* ass on the line.
By all means give people support in doing their jobs, but don't do their jobs for them.
If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.
amen to that
Ignore the haters, they don't understand the politics for this. I used to design industrial Ethernet networks for a large vendor, and we spent quite a bit of time pointing out to customers how dangerous the direct lines were. However, IT departments have very little say over manufacturing networks. This isn't always a bad thing (see the many IT/help desk horror stories). Because the remote access is often required as part of the maintenance contract, offer to partner with manufacturing to install a small firewall with access filters that are controlled by IT, but set (requested) by manufacturing.
A small Cisco ASA, Juniper SRX or its like will do the job nicely, and can shield you from hack attempts along that access path.
The Iranians had airgaps for their centrifuges...
Security is a layered process. Airgaps do help a lot, but then you have to beef up your physical security.
I'll give one example. There was one company that I did an unpaid internship during my college days whose guys gave me a tour. They bragged about their mantraps, their electronic access control mechanisms, and what measures they had in place. I pointed out that the manual override lock on the door was one that was fairly easy to bump, so unless someone is watching the CCTV cameras and sends security at once, an intruder can be in fairly quickly.
They updated those to actual high security locks that have some actual pick resistance.
It doesn't take much to cause a whole data center to go down for a long time (EPO button), so even if an intruder can get five seconds inside a DC, they can cause immeasurable harm to a large company.
One of the best defense measures is segmentation. What machines do the vendors need access to, if you can, put them on their own network segment, firewalled away from everything else. Combine this with limiting outside access.
Not rocket science, but does take time and expense. Good firewalls (Cisco ASA) are not cheap, but they do the job and do it right.
In addition to requiring then to be onsite, negotiate cost of the onsite support in advance. Include a bonus if everything goes according to plan, and if it doesn't, the vendor is covering the extra cost to put it right.
We hear of so many idiots with critical infrastructure connected to the Internet that I felt it my duty to single this post out as #outbreakofcommonsense and say thank you for fighting the good , non-moron fight
Hats off to you sir (or madam as the case may be.)
The I told you so and not our problem method is not what's going to get the right result here. What you've done is passed the blame but not fixed the underlying cause of the problem which is a fundamentally flawed approach to dealing with your systems.
Both the process control team and IT have experience that the other team doesn't have. What is needed is for both teams to sit down and nut out what a site wide acceptable policy for ALL gear is and then have the upper level of management sign off on the policy. This policy should state what is allowed, and what is not allowed, that way management ultimately own the risks.
We have this kind of policy at work. The IT policy comes from the top down and defines not only how the business network is run but also how the process control network is run. There are IT people working in our process control group and there are process control people working in IT to ensure a unified approach.
The fundamental problem here is that the process control guys don't seem to have the security expertise, and the way to fix that is to work together. They should have the facility to allow vendor remote updates. They should have the knowhow to be able to control this link, and the sense and processes to ensure it's not used in an unauthorised way.
> but reality is that IT gets overridden by the Process Control department in a manufacturing business
It happens in a lot of industries. We're forever chasing vendors who think it's ok to pull our systems out from under us to apply updates, sometimes (thankfully rarely) bricking the systems keeping us down until they can make physical repairs.
I don't think there's a surefire technical solution. We disallow access from outside directly to our hardware via our firewall (the best solution -- don't think christmas tree timer, think firewall or switch controls) but since the outsourcing, our firewall is itself under management from an outside group (albeit a different one) and they don't seem to know what they're doing, except to call an operator to press the reset button when a problem is reported.
But the point is, the problem is a social one, not a technical one. I know you haven't had good results so far, but this needs to be fought in management, not in technology. A major production outage gives you fuel -- get riled up, and go talk to some people. Make it plain that the next time the vendor makes any change at all without first approval from a cross-department board, will be the last act that particular vendor does in your company. Put some teeth in your service contract. Hop to it. Your company is at stake.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
The device is called a "firewall" and is set up by an "IT Professional"
You tell the IT guys when (or if) you want that company to be able to connect in remotely. That's it.
Try software development (but not in such a small company that they make you the IT guy on top). It has some of the same problems - you develop some software and for problems connected to that, people will obviously blame you.
But at least you are not the guy who has to try and keep stupidity in general IT use under control. The network intrusion some other guy made possible by mismanaging his equipment is not considered your fault, while IT might get shit for that.
C - the footgun of programming languages
If somebody wants remote access to manage "their" system, these are a few of the things you should insist on:
a) Explicit Contract statements describing the methods of access permitted, when it's permitted and by whom.
b) Contract must spell out the type of testing or diagnostic work to be performed by vendor technicians and who on your company side will pair up with them to validate. The buddy system is to be used at all times when the vendor is gerfinkerpoken mit das machinen.
c) Managed Access, only allow them in on incidents of failure for diagnostic work or for upgrades again, pairing somebody inside your org up with them during the access window. No Carte Blanche access at all, no VPN tokens and no dial up. All access must be over a minimum of TLS 1.1 links, TLS 1.2 is preferred.
d) Penalties in the contract for fucking up. Make it so nasty that if the fuck up they'll pay through the nose if they take you down. Sorry it has to be this way.
e) Specify that you require test results and approval of the test results (including how they tested) before any upgrades. Also provide a test infrastructure or subsystem to allow the vendor to deploy to first to verify that what they're saying actually works. Just because they've done testing doesn't necessarily mean it will work with your hardware and your environment.
f) All workers from the vendor doing the work must be based in your country and be primary to the vendor. No third parties are to have access to your infrastructure or systems. Don't let subcontractors who don't know your systems and processes in.
g) During upgrades or problem sessions a hot call is established to let key stakeholders in your company know what's going on and to provide progress updates on rediation or the success/failure of the situation. Keep your management in the loop and make sure they're aware of the scope of any changes.
h) Backout plans must be provided in case of failures of any changes. I realize this may not be possible with some PLC/Process systems but that should also be a consideration when purchasing these kinds of systems.
i) Maintain air gaps at all times between PLC equipment and any network infrastructure that has access to the Internet or Corporate Intranet. No connections to networks with "office" applications or information flow should be allowed anywhere near your process control networks. The exception to this is when the vendor is troubleshooting or upgrading systems or obtaining log data in accordance with the rest of this.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
Programming a solution to a solved problem is overkill.
In this case I believe it's very appropriate. They have a static arrangement, (vendor wants in, someone turns access on, manually) and when they're done, someone's supposed to shut it off. This process has demonstrated a history of being unreliable. So the solution comes down to one of three things. (1) replace or retrain whoever is in charge of the process in the hopes of improving reliability, (2) automate the process that is not being done reliably, or (3) redesign the process so it's more reliable by default.
(1) is often either futile or short-term. Any number of things can go wrong here, immediately, soon, or long down the road. People get replaced, are out sick for a few days, forget, make mistakes, whatever. (3) is usually unnecessarily expensive, or at least difficult and time-consuming.
It's been my experience that (2) is almost always the best solution. I'm a big fan of automation, and "pick the right tool for the job". (where "tool" refers to either evolved monkeys or computer programming) Computers are almost always more reliable than people, never rely on a person to do a job that a computer can do more reliably. Given the OP's description of the problem, a few minutes of bash or crontab to automate the disabling of the remote access is almost certainly the best answer. I do this sort of thing where I work all the time. I get tired of fixing the same problems over and over that people just can't seem to do reliably. I automate it, and the problem disappears, forever. The initial investment of time always pays for itself. Sometimes in a few days, sometimes in a few weeks. Sometimes once or twice over, sometimes a thousandfold.
Sidenote: whenever something around here breaks, I ask myself a lot of questions. Is there a fair chance it will happen again? Could full automation or manually-initiated scripting have prevented it? Should the system have provided better logging before or during the event? Could the system have predicted the failure ahead of time and given us early warning? Could the system have identified and alerted us of the problem after it occurred, before we (or the client...) discovered it ourselves? Could the system have initiated automated damage control when the failure occurred? This is all a part of automation.
I work for the Department of Redundancy Department.
Parent did mention putting the USB stick in a person. That sounds both painful and inconvenient.
You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...
Umm, IT actually IS a cost center in most companies. My company is a manufacturing firm and I run all our IT operations (among many other duties) and our customers do not pay us a dime for our IT. They pay us to deliver the products we make for them. IT falls in to the necessary/useful expense category. If you are not being paid for the IT then it is by definition a cost center. Lot's of necessary, useful and important things are cost centers - it's not a pejorative. The job of IT is to help the other parts of the business do their job more effectively.
That does not imply that IT is not worth every penny we spend on it. It absolutely is worth every penny and I'd spend more on it if we had the means to do so. It enables us to do things much more efficiently than we would otherwise. If it is well done IT can sometimes be a competitive advantage. Unfortunately a lot of companies don't do a very good job with it.
And guess what: you'll have a new job at Kinkos when the line gets stopped because a trivial update that could have been conducted online is missed because the vendor didn't send someone onsite.
Your job in IT is to support. Not run things. Now be a good IT worker and go change the toner in the LJ4 up on third floor east.
I have never seen anyone fired for doing his job well. I have seen people fired for being insubordinate and abusive (though not often enough!). Are you sure you know the difference?
Yes. I have 14 years of industry experience. I've seen more than you, kid.
#fuckbeta #iamslashdot #dicemustdie