Slashdot Mirror
Ask Slashdot: How Best To Disconnect Remote Network Access?
An anonymous reader writes "Is there a device to automatically disconnect network or otherwise time limit a physical connection to a network? The why? We are dealing with a production outage of large industrial equipment. The cause? The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days. Obviously the main issue is that they were able to do this at all, but reality is that IT gets overridden by the Process Control department in a manufacturing business. They were warned about this and told it was a horrible idea to allow remote access all the time. They were warned many times to leave the equipment disconnected from remote access except when they were actively working with the supplier. Either they forgot to disconnect it or they ignored our warnings. The question is, is there a device that will physically disconnect a network connection after a set time? Yes, we could use a Christmas tree light timer hooked up to a switch or something like that but I want something more elegant. Something with two network jacks on it that disconnects the port after a set time, or even something IT would have to login to and enable the connection and set a disconnect timer would be better than nothing. As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks and repeatedly make blunders like connecting process control systems directly to the internet, use stock passwords for everything, don't install antivirus on windows based control computers, etc. How do others deal with controlling remote access to industrial systems?"
Cant think of anything else...
Tomorrow is another day...
Solves the problem every time.
Enable port security which ties each port to a mac address of the other device connected to it so that all ports on the network switch are locked down to just the devices white-listed to connect. Write down what port your gear is connected to which you want to limit access to the internet, and then simply disable or enable that port to allow it to connect.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
iptables lets you specify times if you're using a linux box as the firewall, otherwise consult the fine manual that came with your equipment or consult a professional with said equipment. This is bog standard.
fucking kidding me.
Time is an illusion. Lunchtime doubly so. ~ Douglas Adams
Set the default gateway to something that only that device uses. Only turn on the default gateway when you want them to access the system ( could be automated by making the default gateway a software service running on an existing machine that you can enable at will ).
There are some firewall/access devices/content filters that restrict access on both time schedules and destination. Maybe talk to your network administrator?
boom goes the dynamite....
Obviously, just put the device behind a firewall. If the firewall operates in bridge mode, it won't use NAT, so the people who insist that their equipment is directly connected to the Internet won't know that it isn't.
The real "Libtards" are the Libertarians!
well hopefully the equipment (and all equipment in the facility aside from the outer firewall) isnt directly connected to the internet.
so then just tell whatever switch/firewall machine that is upstream of the devices not to ever pass packets to/from them that arent
local. and dont give venders access to control your switches/firewalls.
Part of this depends on how they have remote access...is it dial-in? Are they connecting to a jump host via IP connectivity? Is it a VPN? The solution depends on which of those they use, because it's all different. You can use a relay to open/close the actual circuit to the phone line if they dial in; I know a few power companies that use this as a safeguard for their power substations that have dial-up access. If it's a jump host or VPN, then the details of that solution define the approach.
But here's a question for you...what about having a limited time to have remote access would have kept this from happening? From what it sounds like, the process control people would have let them in anyways. And then...what happens if they run out of time, halfway through whatever they're doing? Or even more interestingly, what if they screw everything up (again) but then blame it on being disconnected while they were in the midst of doing something, so they can put the blame on you? This sounds more like a people problem than a technology problem.
For your security, this post has been encrypted with ROT-13, twice.
We usually handle that sort of thing with our Checkpoint firewall (I'm sure other firewalls can do it too but that's what we use). When a vendor needs remote access we put the time limits into the firewall rule. We also make a point of only allowing vendors access from specific IP addresses.
Depending on how your vendors get access you could configure their credentials to only be valid when you want them to be (eg Log on hours in Active Directory)
You could stick an old machine with 2 NICs running Linux in the middle running a simple proxy written in nodeJS. Since you've written the proxy you can write all the rules you want. With node it would be incredibly easy.
I'm a DCS system admin at an oil refinery. We keep the DCS and business lans totally separated, and that directive is driven from the top down. If anyone asks for remote access we just let them know that's NOT going to happen - end of story! It can be a pain getting files from one network to the other (patches, etc.) but certainly worth the effort.
In this case it would have been a futile effort, since they would have called to have the connection made and messed up the upgrade anyhow! If you could trust the production people to do it, it could be as simple as unplugging an Ethernet cable and only plugging it in when it was absolutely necessary. Other wise a network switch can have individual ports turned off remotely. Not an IT that's my son, but this is what I found on a quick look.
your are describing managers who are incompetent and not qualified to run their business.
you have done what you can do. you arent the company president.
this is like dealing with a durg addict. you cant save them. you cant change them. they have to want to change, and they dont want to.
let them fail. let them go bankrupt.
make sure you cover your ass. keep documentation of their stupidity and your warnings.
and keep your resume updated.
Plug the router's power supply into a cheap timer. Twenty bux, problem solved.
* Carthago Delenda Est *
If this system is using an Ethernet connection, just get a Linux or *BSD box running with bridged Ethernet interfaces or pay for a decent smart switch. Heck, you could probably do it in Windows -- that supports bridged interfaces too.
Simply disable the interface connected to the device you want to protect whenever you don't want outside access. With a Linux/*BSD box, this could be accomplished with simple scripts. You'd probably have to write up a simple manual procedure to do it with a switch or Windows box.
I don't know what type of router you have but many do have scheduling capabilities. Actually publishing information like brand and model of router would be pretty dumb.
My first step would be to contact your router manufacturer and if necessary get one that has that capability. You could even put all of your manufacturing equipment behind one unit on it's own segment of the network with limited access from outside, assuming that you really need network access at all.
Unplugging from the network is an option that will permanently take care of the problem.
Non bene pro toto libertas venditur auro
Assuming you have managed switches a simple crontab entry pointing to a shell script can open a connection to the switch an admin down the port that its plugged into. If you want to get really fancy you can have the outbound traffic going via a transparent squid proxy / iptables so you can tell when the port is in use, and keep logs of the connection state.
You can also go with a non-NAT firewall (bridge mode), which will block incoming connections while the device / people on the inside wont know anything is there.
Honestly a timer on an unmanaged switch isn't a bad solution, it takes any technical skill out of the equation, its (assuming the timer doesn't fail) hack proof, and does not require and maintenance / patching to keep secure.
> As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks
If you're going to call someone inept, you better make sure you're not, especially if its your own FUCKING FIELD.
actually, I didn't need to, I've dealt with several cases involving computer crime. Such activity as the OP describes *may* fall under Section 3.
Operation Guillotine is in effect.
How on Earth do YOU get to make fun of other employees at that company? I can think of at least a couple of filtering methods more elegant than a freaking christmas tree timer and I'm not even in IT. If all departments' staff quality is the same as IT I just hope that the "large industrial equipment" is not something that can affect other people.
Filtering access on a per-request basis is one thing, and I see how that's critical and can't think why you haven't implemented this already. Filtering access on a per-timer basis is the WORST WORST WORST idea ever. If I could make that any more caps locked I would. There are SO many things that can go wrong with a blind timer-based disconnection that I won't even bother to list them all, I will just paint the simplest of pictures in a newspaper title: "Incomplete update to a CNC machine leads to hands being sawn off".
Do yourself a favor and change jobs.
I suspect parent is not in the US, so it's not the CFAA. In the UK, it *is* what he called it.
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
Password control? You could implement a policy where all remote connections must be sanctioned by someone on your side and disable/change passwords fatter the remote user notifies you their work is done or after a predetermined amount of time. Bear in mind you should read your support contract before you try this -you may find nasty penalties or be in breach.
I would use an OpenBSD bridge(4). Get a PC with three network ports. Install OpenBSD. Create a transparent bridge between the two networks and use the third connection for access to local ssh(1). I would then configure the pf(4) firewall to allow limited traffic (such as SSH) to cross the bridge. Since the box is a bridge, it's transparent to network traffic and adds an almost negligible latency. Whenever you want to disconnect traffic, log in using ssh (Putty from a Windows box), and turn off the bridge. With an SSH client on a smart phone, you could turn the network off and on from anywhere in the world within a couple of minutes of receiving a phone call. A timeout is easy. Create scripts to enable/disable the bridge and use cron(8), at(1), or a script to fire off the enable/disable scripts at specific times, dates, or intervals.
What Tastecicles says.
Airgap is the only sure way. I know your system isn't classified, but TREAT IT AS IF IT IS.
Inform your boss -- IN WRITING -- that there is to be no internet connectivity for your PCS. If he won't (or higher power's won't let him) agree, then you have to be prepared to either walk, or face the consequences when someone fucks up your system from outside.
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
Follow-up. Again, treating your system as if it was classified...
If you need to apply updates, then get them from the internet, and burn to CD-R and finalize (*not* CD-RW).
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
You do not need to think of any of that if you just stick a managed switch between the internet connection and the equipment.
Enable the port when you want them to have access and disable the port the rest of the time.
Why is this complicated? Why is it a question even?
A Christmas tree light timer ??? How does the OP have a job?
Why is it so hard to only have politicians for a few years, then have them go away?
Either use managed switches to separate the LAN's and lock down the ports to only allow certain mac addresses on the PCS VLAN, or create another LAN with dumb switches. Add a VPN box that is connected to both networks, make it the only allowed method of connecting remotely to the PCS LAN, give the vendor a VPN account, and only enable the account temporarily when it has been approved.
How does the OP have a job?
The real IT guy was fired and replaced with a mid level sales manager. Anonymous submitter for a reason. And maybe they sell Christmas tree light timers and happened to have a few lying around.
“He’s not deformed, he’s just drunk!”
I think the OP is missing something.
I do process control. It's not manufacturing, but that part is irrelevant anyways. The issue at hand is that process control has shifted to control systems that are networked. There are options that don't use ethernet/ethernetIP, but they're increasingly going the way of the Dodo.
We're in a strange time when control systems are increasingly being networked, and the guys that used to do control/automation (and used to do it with relay/hydraulic/pneumatic) don't have the necessary training to integrate the systems correctly. Most IT people don't understand how control systems work and the implications of changing network configurations.
The way forward is to merge IT and process control. Unfortunately, that's easier said than done.
Timer-controlled IP address management can be found in dd-wrt firmware. You can set that machine IP address to have zero minutes and zero seconds of access if you want, and only do it from the router when the vendor calls you to ask to update.
This is literally a $10 ebay WRT-54G with a free firmware upgrade solution.
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
check your contracts before doing any thing you may be on the hook for the full cost of that large industrial equipment after you break the contract
I agree there are too many problems that this could cause (putting the machines on a 'timer') than they would benifit from security. If any company is having mission critical hardware / software handled through the internet you must have engineers or at least senior support staff present to make sure all goes well, after all when are you going to ensure the equipment is running properly, Monday morning 8am?
Have the connection closed with a metal case and a key, when the time comes the company doing the update will contact you and you pay the support staff / engineers to be present. One of the support staff has the key to the lan jack and plugs the cable in, over the phone you cooridinate the service and test the equipment when the work is done. Once all is done disconnect the LAN cable and lock the housing over the jack.
This ensures the right person allows access and is present when the equipment is operated on.
As a bonus you don't find out first shift Monday that production is down.
A loop, by its nature, continues. If that didn't make sense, start reading this sentence again.
You are looking for time-restricted firewall rules.
You can roll them yourself (Linux, Free-BSD), by just having two sets offirewall rules, and switching to the restricted set after the time expires. If you re-inilialize or don't use connection tracking, existing connections get cut. Reloading a firewall rule-set does not cut connections if it does not take too long). You can also isolate this in a specific sub-chain, and then just reload that one to enable or disable the specific connection. That way you have only a low risk of messing up the rest of the firewall configuration.
Better commercial firewalls are offering time-restrictions on rules as well, by basically using the same mechanism.
On the process side, recommend to management that the idiots keeping this access open despite being warned should be fired.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
health Care systems are some times like this in where out side suppliers and vendors have control over systems and some times they don't install updates / say you can't over firewall this system.
...is a post incident review with support people involved, and their management teams, along with directors and executive involvement to identify what the problem was that caused the business to be inoperative for the duration of the incident, what policies and procedures need to be followed going forwards, and so on. Once policies are established, solutions that support those policies can be implemented.
As an example for your situation, since a vendor was involved in an upgrade, that should have been part of a scheduled change. The change should be documented ahead of time as to what is being done, what systems are going to be touched, and who the responsible parties both within the company and external to the company are for that change. Included in the documentation should be the fallback plan for dealing with issues that crop up during and after the change, within an appropriate test window that is included in the change window, as well as clearly defined backout procedures. "fix and fall forward" or equivalent statements are not, and should not be, considered acceptable plans. Wherever possible you want to have documentation attached that the procedures involved have been tested in a suitable test environment. (This may not be possible in situations where a test environment would cost as much to prepare as the production environment.)
As far as limiting remote access, as others have pointed out, such limits are trivial based on what type of remote access is in place, and what policies are established. At the very least account authorizations required for performing changes on production devices should require someone in house approve that authentication, be specific to the time when those changes are scheduled to happen, and should not allow similar access to devices or types of devices not involved in the change.
You never know...
Agreed.
Put in a managed switch, log in the switch to enable / disable the port when you want... <yawn> ...
The best solution is to use this event as a jumping point into securing it right... No matter what technical solution you come up with, the weakest link are the people. Education, some firings, and getting a better vendor are the real next step. Remote access can be a marvellous tool to getting problems straightened out without flying people in, but it sounds like these are the kind of people you wouldn't let walk unescorted in the plant...
More Caffeine. NOW
Plug the switch into a web controlled power switch:
http://www.digital-loggers.com/lpc.html
Eight power jacks that can be independently controlled over your network. You can control access to the entire device or individual sockets with multiple users and passwords, and they have built-in scripting functionality that shut off sockets based on the time, power-cycle if a repeating ping test ever fails to get a response, and other options I haven't bothered to look into. A real party. I think they're about $100.
What you want to do is SO frigging simple it's embarrassing. So, what this tells me is that you are not spending money on IT staff. Quit being a cheap bastard and hire at least ONE competent IT person.
Your thin skin doesn't make me a troll
The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days.
This isn't a technology problem.
Through their incompetence, they caused damages. Collect your evidence, hire a lawyer, and make demands. If they refuse to pay, sue them.
Watch how fast they start caring about doing remote upgrades more carefully, competently, and with customer involvement. The only thing companies collectively care about is making money. At the very least, you'll cause their liability insurance rates to go up.
Please help metamoderate.
Cut a network cable and break the wires out, one to each port of one of these:
http://www.sainsmart.com/8-channel-dc-5v-relay-module-for-arduino-pic-arm-dsp-avr-msp430-ttl-logic.html?___store=en&___store=en
Then, connect it to an ATtiny - ATtiny85 should be fine; it's only a couple of bucks - and program it to go on and off as you wish. Side benefit: it runs on batteries in case the power goes off.
If you submit the idea to Make, I get credit.
All Internet connections must cross a Firewall. Disable inbound connections, done.
Seriously, this is a question?
You wouldn't have an osm to worry about.
~Sticky
A Christmas tree light timer ??? How does the OP have a job?
You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...
I just got done with a contract at a large bank (It's one of the 50 largest companies in the United States)... all their deployments are run off USB drives hung off servers at their retail locations, they have 512kbit backhauls to their corporate locations, run DHCP over the WAN, have no QoS, and I kid you not -- about 5% of the managed switches have been forced to 10mbit half-duplex.
And since they're so security conscious, all the workstations have drives that are encrypted, have antivirus that runs every 4 hours, whether you're using the system or not, a couple other "intrusion detection" apps that also run, sometimes on overlapping schedules, sometimes when trying to patch the operating system... and for the bonus round: An account used for software installation that has full local admin to every workstation... and has a password that's the same as the account name.
-_- Attaching one of those appliance timers to a switch to shut it off at predefined intervals seems so stupidly obvious, but when you realize how stupid the average person is, and then realize that the ones stupider than that work in HR and Accounting, you quickly conclude the same thing the rest of us in this industry have:
Just drink your damn beer and try to drown out the stupid. Thinking about it will only depress you. Trying to do something about it will get you fired. Trust me... there is no faster way to get fired in IT than doing your job well... because you'll get noticed by all the incompetent asshats that HR and Accounting let in, and they'll form an alliance against you to get rid of you. And for the super jaded special bonus round... trying to get shit done will make you realize that the reason you can't get anything done is because everybody has silo'd themselves away with crucial documentation, settings, or knowledge, to assure themselves of continued employment. Start poking around, and they'll feel threatened. When they feel threatened, they'll find some way to go behind your back and make you look bad. Do this enough times and management will consider you an agitator and... ker-chop.
If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.
#fuckbeta #iamslashdot #dicemustdie
That's fine, until the process control guys unplug from your nice managed port, run a cable across the floor and plug into a port that you're not actively managing. And they will do that. If you don't think so then you haven't worked in that type of environment.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
Configure your firewall rules so that access to these systems has to come from the internal network or through a VPN.. Create a VPN account for the vendor but leave it disabled. If there is a legitimate need to access the system they can submit a request through your internal change control process (or have an internal contact submit the request on their behalf) and pending approval the VPN account can be enabled when the work is to begin and disabled again once the work is complete. In fact if you are subject to SOX, HIPPA etc. you are probably required to grant external access this way.
You can probably do the same with firewall rules rather than a VPN but personally I don't like granting external access to internal network resources.
Any insufficiently advanced magic is indistinguishable from technology.
This isn't my field, but I think you should do nothing. IT's job is to provide network access. Process Control's job is to keep the machinery running, and if they fail to do so despite your warnings, it's their ass on the line.
Yes, "not my problem" is a classic way to make a workplace awful, but consider this: if Process Control can't get a software update to their machinery because you've blocked it, and something bad happens (worst-case scenario, a machine kills someone), then it's *your* ass on the line.
By all means give people support in doing their jobs, but don't do their jobs for them.
If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.
amen to that
Agree. They will do this. Seen it everywhere. And if the run is too long for a cable, prepare for wireless.
~Sticky
Airgaps aren't a panacea. USB keys, CDs, even floppy disks (yes, these places still have those) can all bridge an airgap in a non-detectable manner.
Most of these systems have no actual monitoring to ensure that the integrity of the network stays constant. And, if it makes a process control professional's life easier, they WILL connect it to the internet for 'a little while', go home, forget, and completely deny they did it if the fit hits the shan.
The people need change too.
~Sticky
'cuase I got no real mod points...
Your thin skin doesn't make me a troll
Lots of people have been recommending proxies. Why not put it with the equipment, and control physical access there? If they want to plug it in, they're plugging into your tiny micro computer which is now integrated into the equipment.
That, or you might find an el cheapo four port switch between a production machine and its normal port where there wasn't one before, so unless you set the max MAC addies to one as a matter of daily business, you may have an ugly surprise.
Probably cheaper than a hi-falutin' IT solution; you can explain when to pull the plug, when to put the plug in;
he can do all sorts of useful things around the place; fetch the burgers or donuts; did I say its cheaper? And
in a few years, he can probably run the IT department with a big pay raise, and in turn hire another Mexican
kid.
Ignore the haters, they don't understand the politics for this. I used to design industrial Ethernet networks for a large vendor, and we spent quite a bit of time pointing out to customers how dangerous the direct lines were. However, IT departments have very little say over manufacturing networks. This isn't always a bad thing (see the many IT/help desk horror stories). Because the remote access is often required as part of the maintenance contract, offer to partner with manufacturing to install a small firewall with access filters that are controlled by IT, but set (requested) by manufacturing.
A small Cisco ASA, Juniper SRX or its like will do the job nicely, and can shield you from hack attempts along that access path.
The Iranians had airgaps for their centrifuges...
Security is a layered process. Airgaps do help a lot, but then you have to beef up your physical security.
I'll give one example. There was one company that I did an unpaid internship during my college days whose guys gave me a tour. They bragged about their mantraps, their electronic access control mechanisms, and what measures they had in place. I pointed out that the manual override lock on the door was one that was fairly easy to bump, so unless someone is watching the CCTV cameras and sends security at once, an intruder can be in fairly quickly.
They updated those to actual high security locks that have some actual pick resistance.
It doesn't take much to cause a whole data center to go down for a long time (EPO button), so even if an intruder can get five seconds inside a DC, they can cause immeasurable harm to a large company.
One of the best defense measures is segmentation. What machines do the vendors need access to, if you can, put them on their own network segment, firewalled away from everything else. Combine this with limiting outside access.
Not rocket science, but does take time and expense. Good firewalls (Cisco ASA) are not cheap, but they do the job and do it right.
This is a problem with the organization of your business and cannot be fixed with a technical solution.
"the main issue is that they were able to do this at all, but reality is that IT gets overridden by the Process Control department in a manufacturing business. They were warned about this"
You need to raise the risks to management and if they choose to override your recommendation then the problem is theirs, not yours.
blindly antisocialist = antisocial
We hear of so many idiots with critical infrastructure connected to the Internet that I felt it my duty to single this post out as #outbreakofcommonsense and say thank you for fighting the good , non-moron fight
Hats off to you sir (or madam as the case may be.)
You said they need remote access and botched the upgrade. There is no technological solution for this. If your management insists an incompetent vendor have such access and thinks you can do something about this result then the solution is to find another job. If you aren't being told to fix it, realize that it's not your problem.
You can't fix stupid; you either have to accept it or escape it.
> At the least, a civil suit for any damage caused.
Good point. I assume there is a contract, and the contract specifies damages caused by incompetence? In that case it would not be the OP's problem again. The question is whether it is worth suing a supplier - usually that does not exactly help the business relationship. But it may be leverage to come to some kind of resolution.
The I told you so and not our problem method is not what's going to get the right result here. What you've done is passed the blame but not fixed the underlying cause of the problem which is a fundamentally flawed approach to dealing with your systems.
Both the process control team and IT have experience that the other team doesn't have. What is needed is for both teams to sit down and nut out what a site wide acceptable policy for ALL gear is and then have the upper level of management sign off on the policy. This policy should state what is allowed, and what is not allowed, that way management ultimately own the risks.
We have this kind of policy at work. The IT policy comes from the top down and defines not only how the business network is run but also how the process control network is run. There are IT people working in our process control group and there are process control people working in IT to ensure a unified approach.
The fundamental problem here is that the process control guys don't seem to have the security expertise, and the way to fix that is to work together. They should have the facility to allow vendor remote updates. They should have the knowhow to be able to control this link, and the sense and processes to ensure it's not used in an unauthorised way.
That's not an "anarchist quote", it's a quote from 1984, a book about totalitarianism. How dumb do you have to be to think that the only alternative to totalitarianism is lawlessness and anarchy? And even if the guy were an anarchist, anarchy rejects hierarchical, centralized state power, not voluntary associations and self-governance.
> but reality is that IT gets overridden by the Process Control department in a manufacturing business
It happens in a lot of industries. We're forever chasing vendors who think it's ok to pull our systems out from under us to apply updates, sometimes (thankfully rarely) bricking the systems keeping us down until they can make physical repairs.
I don't think there's a surefire technical solution. We disallow access from outside directly to our hardware via our firewall (the best solution -- don't think christmas tree timer, think firewall or switch controls) but since the outsourcing, our firewall is itself under management from an outside group (albeit a different one) and they don't seem to know what they're doing, except to call an operator to press the reset button when a problem is reported.
But the point is, the problem is a social one, not a technical one. I know you haven't had good results so far, but this needs to be fought in management, not in technology. A major production outage gives you fuel -- get riled up, and go talk to some people. Make it plain that the next time the vendor makes any change at all without first approval from a cross-department board, will be the last act that particular vendor does in your company. Put some teeth in your service contract. Hop to it. Your company is at stake.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
The device is called a "firewall" and is set up by an "IT Professional"
You tell the IT guys when (or if) you want that company to be able to connect in remotely. That's it.
Who is more inept? The inept or the inept who hired himt? ~Obiwan (while working IT)
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
"First things first -- but not necessarily in that order"
-- The Doctor, "Doctor
-keyboar battt dead
This Ask Slashdot has to be in the top 10 worst Ask Slashdots...
Thanks for the pep talk Danny Downer. Whats a person that loves computers supposed to do then; drive taxis and live off food stamps?
"Is there a device to automatically disconnect network or otherwise time limit a physical connection to a network?"
Yep, it's called a router.
Seems to be something that needs to, at least in part, be governed by a contract with the supplier. You might find yourself the scapegoat if off your own back you buile q Wallace & Gromit inspired steam-powered Kill'O'Network contraption.
-- Using the preview button since 2005
ELEGANT! ELEGANT!
Sheesh. IPTables, Cisco timeout functions...no imagination whatsoever.
Write an app for a dedicated iPhone (hooked to a power adapter) that is interfaced with an RJ45 switch (low-voltage control circuit for a small relay in place of the usual rotary switch) for the ethernet cable , so that the vendor has to call that iPhone and maintain a connection to keep the ethernet connection active. Limit the phone number sharing to vendors only.
"Something with two network jacks on it that disconnects the port after a set time"
No wonder some people have a hard time finding jobs, since a lot seem to be taken by cheap labor with unrelated or irrelevant knowledge or workforce repurposed from other departments to cut the costs of hiring someone who has the proper knowledge. I am not in IT, never was, never will, yet even I know of the device the poster seeks with multiple jacks and connection handling functions, which are magic boxes brought by blue fairies in the middle of the night and are called... wait for it... MANAGED SWITCHES!
I mean come on, really?
If you really don't know what to do, then at least run a google query with managed switch session timeout or vpn router session timeout.
I am putting myself to the fullest possible use, which is all I can think that any conscious entity can ever hope to do.
Why do you want to disconnect after a set time interval? Your time will never be correct. Sometimes, legit maintainance might take longer (e.g. if they're monitoring the system to trace a problem) and often it will be much shorter.
Without knowing the details, what you need is a point inbetween that IT Security controls, and the procedure that says enabling remote access requires form 123a filled out. Which would be a simple paper saying "please enable remote access for vendor X on (datetime) until (datetime)".
Everyone will hate you for the added bureaucracy, but this is the one and only way to guarantee that no outside vendor can access your system without your knowledge.
Take care regarding the wording of the form. Nobody likes the beg IT to do anything. It should sound like an order, that'll make them feel a lot better, and you're not planning to deny any of these anyways.
Assorted stuff I do sometimes: Lemuria.org
If you're serious about finding work, try moving to a state that's (mostly) the opposite political persuasion. It's never black and white, so you'll quickly find you have natural allies against the common enemy.
Oh, and quit reading slashdot cold turkey. That's part of your problem.
Try software development (but not in such a small company that they make you the IT guy on top). It has some of the same problems - you develop some software and for problems connected to that, people will obviously blame you.
But at least you are not the guy who has to try and keep stupidity in general IT use under control. The network intrusion some other guy made possible by mismanaging his equipment is not considered your fault, while IT might get shit for that.
C - the footgun of programming languages
I'm not sure whether it's more stupid to allow remote access to vendors or to let it make updates/upgrades on live systems.
In my poor region we mandate a real test system on which any change needs to be challenged first along with rollback procedure.
Only then we go for the real change.
But we live in a poor region and cannot afford SEOs (Stupidity Executive Officers), only technicians are allowed.
Sent as ripples into the electromagnetic field. No single photon has been harmed in the process.
Here's a thought. If firewall/router configs are too fluffy for the organisation to grasp[1] then use a technology that the end-users are familiar with. For example a router with a power supply controlled by a key switch. Who has the key, when it went out, why do you want it?, how long, what will affect? can be managed by production/shift managers as other things. It also means that there is some bod on site who has to be told by the vendor what's supposed to be happening and gets permission for *that thing* with *certain risks* who is on site and shares responsibility for the tweaking. If the vendor turns up on site then they jack-in to the LAN with their laptop with your firewall controls etc and the LAN connects to the process system via the switched router, and the same chain of *responsibility* applies with the shift-manager carrying the can and so exercising proper supervision.
1- you're incompetent.... what you want to do is networking 101
2- you're an idiot. your job is to help your users, not to piss them off
3- you fail to realize you're having a political/organizational problem, not a technical one
4- the solution you're contemplating is medieval
Honestly, you sound like a gleeful entitled bitch thay just stumbled.on some blackmail material.
The Cloud - because you don't care if your apps and data are up in the air.
1. Managed switches
2. VPN
3. Firewall
I would choose 2 and/or 3 tied to a request/approval process that requires authorization, justification, and duration.
Keep the Classic Slashdot.
Take your homework to stackexchange timmah.
the whole of the problem is with the server end anyway - what's the difference between allowing someone access for an hour, and allowing them access for a day? Or a minute?
The server side either has security controls on it, in which case it doesn't really matter how long someone's connected; or it doesn't have adequate security in which case a 1 minute connection is sufficient to blast a heap of viruses at the server.
Seems to me the poster is just a whiny fool who is inept himself.
If somebody wants remote access to manage "their" system, these are a few of the things you should insist on:
a) Explicit Contract statements describing the methods of access permitted, when it's permitted and by whom.
b) Contract must spell out the type of testing or diagnostic work to be performed by vendor technicians and who on your company side will pair up with them to validate. The buddy system is to be used at all times when the vendor is gerfinkerpoken mit das machinen.
c) Managed Access, only allow them in on incidents of failure for diagnostic work or for upgrades again, pairing somebody inside your org up with them during the access window. No Carte Blanche access at all, no VPN tokens and no dial up. All access must be over a minimum of TLS 1.1 links, TLS 1.2 is preferred.
d) Penalties in the contract for fucking up. Make it so nasty that if the fuck up they'll pay through the nose if they take you down. Sorry it has to be this way.
e) Specify that you require test results and approval of the test results (including how they tested) before any upgrades. Also provide a test infrastructure or subsystem to allow the vendor to deploy to first to verify that what they're saying actually works. Just because they've done testing doesn't necessarily mean it will work with your hardware and your environment.
f) All workers from the vendor doing the work must be based in your country and be primary to the vendor. No third parties are to have access to your infrastructure or systems. Don't let subcontractors who don't know your systems and processes in.
g) During upgrades or problem sessions a hot call is established to let key stakeholders in your company know what's going on and to provide progress updates on rediation or the success/failure of the situation. Keep your management in the loop and make sure they're aware of the scope of any changes.
h) Backout plans must be provided in case of failures of any changes. I realize this may not be possible with some PLC/Process systems but that should also be a consideration when purchasing these kinds of systems.
i) Maintain air gaps at all times between PLC equipment and any network infrastructure that has access to the Internet or Corporate Intranet. No connections to networks with "office" applications or information flow should be allowed anywhere near your process control networks. The exception to this is when the vendor is troubleshooting or upgrading systems or obtaining log data in accordance with the rest of this.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
Yes, we could use a Christmas tree light timer hooked up to a switch or something like that but I want something more elegant.
http://en.wikipedia.org/wiki/KISS_principle
In looking for something elegant you are going to make it more complex and increase the probability that it will fail.
Graybar has some nice external wiring boxes, very suitable for locking down external connections. I've often recommended similar boxes, the outdoor electrical outlet boxes with covers, for use in small data centers. It helps protect electrical outlets of critical equipment plugged into the wall from being accidentally tripped over or casually unplugged.
Use 802.1x authentication on the switch ports and you can control access anyway you want.
http://www.juniper.net/us/en/local/pdf/whitepapers/2000216-en.pdf
http://www.cisco.com/en/US/docs/switches/lan/catalyst2950/software/release/12.1_9_ea1/configuration/guide/Sw8021x.html
Now I hope and pray that I will But today I am still, just a bill
http://www.aviosys.com/9258st.html
If you use a controller that supports time limits for an outlet being energized, you can then use that outlet to segment your network. You would do that by having that outlet power a switch or hub that would attach the external network and the internal network.
The advantage of that type of setup is that you have access any time that you need it, but you must actively acquire that access. If you don't trust your supplier with access to the switch, you guarantee that they have access only when you grant it to them.
Programming a solution to a solved problem is overkill.
In this case I believe it's very appropriate. They have a static arrangement, (vendor wants in, someone turns access on, manually) and when they're done, someone's supposed to shut it off. This process has demonstrated a history of being unreliable. So the solution comes down to one of three things. (1) replace or retrain whoever is in charge of the process in the hopes of improving reliability, (2) automate the process that is not being done reliably, or (3) redesign the process so it's more reliable by default.
(1) is often either futile or short-term. Any number of things can go wrong here, immediately, soon, or long down the road. People get replaced, are out sick for a few days, forget, make mistakes, whatever. (3) is usually unnecessarily expensive, or at least difficult and time-consuming.
It's been my experience that (2) is almost always the best solution. I'm a big fan of automation, and "pick the right tool for the job". (where "tool" refers to either evolved monkeys or computer programming) Computers are almost always more reliable than people, never rely on a person to do a job that a computer can do more reliably. Given the OP's description of the problem, a few minutes of bash or crontab to automate the disabling of the remote access is almost certainly the best answer. I do this sort of thing where I work all the time. I get tired of fixing the same problems over and over that people just can't seem to do reliably. I automate it, and the problem disappears, forever. The initial investment of time always pays for itself. Sometimes in a few days, sometimes in a few weeks. Sometimes once or twice over, sometimes a thousandfold.
Sidenote: whenever something around here breaks, I ask myself a lot of questions. Is there a fair chance it will happen again? Could full automation or manually-initiated scripting have prevented it? Should the system have provided better logging before or during the event? Could the system have predicted the failure ahead of time and given us early warning? Could the system have identified and alerted us of the problem after it occurred, before we (or the client...) discovered it ourselves? Could the system have initiated automated damage control when the failure occurred? This is all a part of automation.
I work for the Department of Redundancy Department.
I would hook up an Arduino to this (https://www.sparkfun.com/products/10747) and use it to control power on the internet router. It is not a physical disconnect, but it is probably the next best thing.
You can up the Arduino to listen via Ethernet, look for card swipes, check for physical presence, however you want to control it.
cej102937
You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...
Umm, IT actually IS a cost center in most companies. My company is a manufacturing firm and I run all our IT operations (among many other duties) and our customers do not pay us a dime for our IT. They pay us to deliver the products we make for them. IT falls in to the necessary/useful expense category. If you are not being paid for the IT then it is by definition a cost center. Lot's of necessary, useful and important things are cost centers - it's not a pejorative. The job of IT is to help the other parts of the business do their job more effectively.
That does not imply that IT is not worth every penny we spend on it. It absolutely is worth every penny and I'd spend more on it if we had the means to do so. It enables us to do things much more efficiently than we would otherwise. If it is well done IT can sometimes be a competitive advantage. Unfortunately a lot of companies don't do a very good job with it.
Agreed, they aren't a panacea. They're part of a layered mechanism. However, an airgap would COMPLETELY solve this guy's issue -- the vendor pushing an unwanted software/firmware change over the 'Net.
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
And they will do that. If you don't think so then you haven't worked in that type of environment.
I see you have. :)
So, this gets to the heart of the matter. The OP needs a meatspace policy, not [just] an ACL. That policy needs to specify consequences for routing around company security. The best time to get this through is before the critical system comes back online.
If the management is unwilling to give him that policy, then he needs to let them know what will happen, and preferably get a paper trail ACK on that. If management still wants to make such outages his problem, then he needs to find a non-abusive employer.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Yes, it's called "a firewall" They do many very useful things. Allow them access on a port, and have a script running that disables it after a time.
Lock MAC addresses to a single port and/or firewall everything that IT can't maintain control over.
I am becoming gerund, destroyer of verbs.
The latest seem to be terminal servers (e.g. Oracle Secure Global Desktop) with time managed access to certain services.
Vendor calls customer, customer grants access for a specific user and system for a limited time, all on screen actions are recorded.
In the past more funny things existed. Some vendor support was by modem dial in only (e.g. EMC), so the customers had a switch that connected/disconnected the physical phone lines to prevent unwanted access.
Configure your VPN headend to authenticate against a RADIUS host that is configured for a one-time-password. You must provide that one-time-password to the vendor each time they wish to connect. The second time the same password is used, it should be denied. This should NOT be a token-derived password they posses, but rather something they must get from you over the phone after authenticating themselves in some other way.
Ensure that the connection has a timeout of some reasonable time that won't kick them out of a legitimate activity.
$ man woman *
-bash:
Hahaha.
Yeah f-ing right.
Get a $1600 plane ticket to nowhere, overbooked, because you want it now. now now now.
Drive onsite to your craphole kingdom out in the corn fields.
Walk to the boring front desk and wait. Pissed off whoever is like "why didn't you call about this visit ahead of time"
(even though I did, 7 out of 8 people know I'm going to be there... you just happen to be the 8th.)
Watch your shitty safety videos. Listen to "how troublesome machine is" speech.
Sign off on your non-disclosure sheets for your "top-secret" process that everyone else in the industry is doing.
Wait for your machine operators to stand around until their shift ends, meanwhile the machine is doing nothing. I'm doing nothing.
Can I make the chang-- nope, were' in production! But you're not doing anything! Nope, not enough time to run new job...
It's all about the feelings of control in your otherwise pathetic life, to make me stand there for hours as "punishment". Sick bastards.
Because we failed to predict the "mission critical" thing that was never mentioned or tested. Even if asked as part of the sale (what does it need to do).
(Can you provide test product? Nope, costs too much to ship. oh by the way we need the machine 3 weeks early. and give us a discount.)
(lots of time to think about this disaster of a project).
Finally at about midnight connect to the machine and download a change that takes 3 minutes, from a laptop that they won't even give an extension cord to power.
Be like... okay, where's the test product? hello? skeleton staff just smart enough to turn off the lights and air compressor.
Pack up and leave.
Phone call at 5 a.m.
Whining about how the update (that impossibly) changed this or that. Update was some very specific very narrow change. Just some stupid misaligned sensor or setup error it's assumed.
Race back in at about 6:30 am because nearest hotel is godawful far from plant.
Constant phone calls of "we are down". (burn in hell for leaving 6 messages for somebody already coming to your place).
Oh look, you forgot to turn air on to the machine. and look, here's the message saying you forgot to turn air onto the machine.
Questions about why your update "broke" the machine.
"You know how much this downtime costed the company???"
It's hell to be a machine manufacturer.
The pay isn't even that great.
And IT is just another obstacle, a gatekeeper and an entity of "NO!" instead of trying to be part of the solution.
if only they didn't have that rule in place about moderating in a discussion you're involved in...
Operation Guillotine is in effect.
The Iranians had airgaps for their centrifuges
They say the only computer that can't be hacked is one that unplugged... but when your adversary is (apparently) a U.S. or Israeli Intelligence Agency, even that's not going to save you.
-
- - You can't take something off the Internet! That's like trying to take pee out of a swimming pool.
Wouldn't almost any managed switch with a command line interface or SNMP do this? You could remotely and on a scheduled basis tell the switch to enable or disable traffic to the port that your equipment is connected to.
No ethernet, no internet, no remote access.
Article X: The powers not delegated... by the Constitution...are reserved...to the people
There are two faults here. One is yours in that you left your equipment connected to the network such that external access was allowed. The other is the supplier/vendor of the equipment who performed a modification of the equipment without your explicit approval. IMO, the supplier is fully responsible/liable for ALL of your costs and/or losses caused by this action on their part. I think an attorney would agree, unless there is a clause in your support or purchase/license agreements that allows for this, in which case, caveat emptor! As for a network connection timeout, this is not really feasible. However, proper configuration of your network firewalls (I assume you have such?) should mitigate this sort of unauthorized access. If the equipment doesn't need local network access for management/monitoring purposes, then simply disconnect it. If it does, then the firewall rules have to be adequate to block remote access without your permission and intervention.
Sometimes, real fast is almost as good as real-time.
I have never seen anyone fired for doing his job well. I have seen people fired for being insubordinate and abusive (though not often enough!). Are you sure you know the difference?
Yes. I have 14 years of industry experience. I've seen more than you, kid.
#fuckbeta #iamslashdot #dicemustdie
Quite the rant. Having been one of the fucktard IT drones for a decade, I can tell you that the feeling was probably mutual. It's more likely than not that the IT guys you worked with didn't have a lot of choice about what patches got pushed when, especially when production guys claim to be too busy/important to create a test environment to roll out the patches to first. On the other hand, there is also no shortage incompetent IT guys.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
A Christmas tree light timer ??? How does the OP have a job?
We use exactly this solution for the public WiFi outside our cafeteria, it was an almost free solution and it keeps anyone from doing something like downloading porn or hacking someone from that connection outside of the few hours a day it should be enabled when there are people around. Newer routers can enable and disable SSID access on a schedule but we were working with a unit without such a feature, plus the timer is more green =)
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
www.AddaBazz.com It Is New Social Site in World
... a necessary expense you should spend as little as possible on. Unfortunately, that's what they teach MBAs and the like in college.
You are making a biased assumption not based in any actual fact. I assure you that is not taught in any MBA curriculum from first hand experience. I have an engineering degree but I also have a business degree. I'm an engineer but I also am a certified accountant. They do NOT teach that IT is the same thing as paper towels in business school. I know because I've been through the classes myself. In fact they actually spend a surprising amount of effort trying to teach how IT can be a productivity multiplier. You would be surprised how many people who go to study business management are from IT. Probably 10-15% of my class had some form of IT background. They were there to learn how to more effectively manage which involves more than just optimizing IT.
Now there are managers who don't really grok IT and do regard it as an extravagance. Sounds like you've run into some of these fools. My sympathies for that. On the other hand there are plenty of IT people who greatly overestimate the importance of what they do and think that money should be poured endlessly into IT without any concept of the effect on the rest of the business. That door swings both ways. The trick is to find that nice middle ground where IT can be a real asset without costing the company needlessly.
Create an adapter that physically transforms RJ-45 to some obscure connector format, then epoxy that into the ethernet ports on the devices. Then you keep the adapter. Anytime someone wants to connect the device to the network, they have to come track down the adapter from you.
Realistically, just produce documentation showing you warned IT that this would create a shitstorm and take it to the big boss to show how IT dropped the ball. The problem should solve itself at that point.
If you want to block the manufacturer from access to your machines, put them behind a firewall (I like Linux + iptables, because it's cheap), then put the right filters so they can't bloody get to the machine.
You do /not/ want a christmas tree timer providing access for N minutes, etc. Because in the digital age, it likely takes much less than N to trash your equipment.
-Ken
...and solve it with hardware. The tricky bit is getting the vendor to put their fingers on the anvil right before you smack them in the knuckles with a ball peen hammer.
Now, make sure that you can connect to your department's firewall from the outside world.
Now, reconfigure the firewall so that it blocks all access except to/ from your office, and make that the machine's default route.
If the vendor wants to do something to the machine, they can do it from your office.
You may need to fire the first person to disconnect the machine from the firewall, and publicise the fact ; you may need to hide the firewall in a locked IT cabinet ; you may need to monitor the state of the physical connections. These are details that a competent IT person should be used to dealing with.
When the vendor wants to fuck with the machine again, you do your QC on their procedures before letting them even think about accessing the real machine.
Personally, I'd use the super glue and side cutters.
Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"
This is not a technical problem, it's a political one. I know this doesn't actually answer your question, but this needs to be said.
In your own post, you told us the answer to the problem: leave equipment disconnected unless absolutely necessary. What you want is a solution to bypass best practices without doing things completely the wrong way. You're really asking for a lock that will keep a door half-open without letting it open completely. (What's to stop the vendor from applying a crappy update during a time-limited session? What happens if the update process gets cut off half-way though? Now your equipment is completely bricked and they can blame you for tampering with their connection! At the very least, it will create a shit storm for your legal department; at the most, it will leave your company footing the bill to fix the system.)
I also think that you haven't considered that this solution probably won't work anyway. You have another department ACTIVELY SUBVERTING YOU! Do you think that the process folks are suddenly going to back off when you start killing their network connections? Maybe you could play dumb for awhile, but once someone figures out what you're doing (or convinces a higher-up to force you to 'fix the problem') it's going to blow up in your face. If you think that you have political problems now then just wait until your trickery comes to light. In any case, they're going to demand that you give them "time limited" sessions that run for 24 hours or even longer. Also, they might just use their political clout to force you to remove any restrictions that you put in place anyway.
The ideal solution (which you're already tried) is to educate the process department on the best practices and encourage them to implement them. However, if that's not going to work then you need to resort to plan B. Inform them of the risks in no uncertain terms (this also seems to be something that you've done). Do this often. Send them memos, send them emails, etc. and save their responses. If they send you an email telling you that they're not going to secure their network, then save that email. (You work in IT, so I don't have to tell you to keep multiple copies of this email in many different places.) This way, when shit hits the fan, you can point to all the warnings you gave and thus save your ass in the ensuing investigation.
-1 disagree is not a modifier for a reason. -1 troll, flaimbait, redundant, overrated are NOT acceptable substitutes.
In the Linux world we call it scripting the iptables. What you want to do is rather easy.
The constraints of a REAL productin environment can be many. It's easy for you "real IT guys" to say he shouldn't have a job, but you've no clue about the constraints he operates under. I have an active MD and an active OpD above me in my company and regularly have to fight to avoid having our network integrity compromised by simiarly stupid decisions including; 1. Allowing a customer direct access to our SQL server - access to views and/or via web services wasn't acceptable to them. Problem here was our MD is talking to senior procurement managers in the customer company. Thankfully I have a personal friendship with the customer company's IT guy, so when I phoned him we worked it out that a simple web service would suffice; 2. Allowing a vendor to connect critical equipment directly to the net bypassing the firewall 3. Allowing a vendor root access to our domain The truth is I don't always win the argument, so my environment also had a few stupid and dangerous elements. Another truth is I'd never make the mistake this guy did to ask for help on here. Most of you "real IT guys" haven't got a clue what it can be like in a real production environment, so you just dispense smart ass sarcasm with no other purpose than to inflate your own egos at the expense of the poor guy who, lets face it, must already be under more than enough pressure and humiliation without your contributions
So, so true. I was a consultant for several years, then shit happened that was really all I needed for an out. I still love computers but there's no fucking way I'm going back into the industry,
Operation Guillotine is in effect.
it's like going into business with your best friend: bad fucking news.
Operation Guillotine is in effect.
When General Hammond says "I'm the boss, make that thing spin!", you take that thing and make it spin! Don't tell him that it doesn't actually spin, you'll be outsmarting him and PHBs don't like that.
Operation Guillotine is in effect.
Whoever allowed that is a freakin' moron.
If you're allowing a vendor to update your production stuff whenever they feel like it and not coordinating with you, then whoever decided on that within your company is a bloody idiot who deserves this.
I feel your pain, but you (in addition to making sure this doesn't happen again) should be making damned sure to scream loudly that you told them so.
If you blindly trusted your vendors and game them that kind of access, something has gone horribly wrong at your company.
This sounds like a case of being lazy and stupid instead of really thinking about your business needs.
Lost at C:>. Found at C.
You do not say what your place is in the organization. This will directly affect what the proper answer is to your query. From the hip, it sounds like you are some IT minion and that you are attempting to vastly overstep your authority.
If you are management and have the authority to make such a decision, you merely tell the network folks to set up a rule for the firewall or router to simply disallow access except at specified times. This rule could be automated easily but your network folks will know how to do that.
If you are not management, you do whatever management tells you to do. You could tell them, "I told you so", if you had warned them about this possibility previously but that lacks political/social grace. Ideally speaking, you would bring up the possibility of restricting access for the vendor to management. If you are an IT minion asking how to implement this, you are in the wrong business.
Regardless, everything that happens, occurs because management set it up that way. It is not your place to steer the business. At best, it is your place to make recommendations... but even then, management is not required to follow your recommendations. If management is doing stupid things that you know will make the company crash, your only recourse is to start looking for another job before your current one goes away due to the company failing. Correct and incorrect is defined purely by management.
Good luck.
"Someone needs to talk to the tree of liberty about its ghoulish drinking problem." by ohnocitizen