Slashdot Mirror
Ask Slashdot: How Best To Disconnect Remote Network Access?
An anonymous reader writes "Is there a device to automatically disconnect network or otherwise time limit a physical connection to a network? The why? We are dealing with a production outage of large industrial equipment. The cause? The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days. Obviously the main issue is that they were able to do this at all, but reality is that IT gets overridden by the Process Control department in a manufacturing business. They were warned about this and told it was a horrible idea to allow remote access all the time. They were warned many times to leave the equipment disconnected from remote access except when they were actively working with the supplier. Either they forgot to disconnect it or they ignored our warnings. The question is, is there a device that will physically disconnect a network connection after a set time? Yes, we could use a Christmas tree light timer hooked up to a switch or something like that but I want something more elegant. Something with two network jacks on it that disconnects the port after a set time, or even something IT would have to login to and enable the connection and set a disconnect timer would be better than nothing. As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks and repeatedly make blunders like connecting process control systems directly to the internet, use stock passwords for everything, don't install antivirus on windows based control computers, etc. How do others deal with controlling remote access to industrial systems?"
Cant think of anything else...
Tomorrow is another day...
Solves the problem every time.
Enable port security which ties each port to a mac address of the other device connected to it so that all ports on the network switch are locked down to just the devices white-listed to connect. Write down what port your gear is connected to which you want to limit access to the internet, and then simply disable or enable that port to allow it to connect.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
iptables lets you specify times if you're using a linux box as the firewall, otherwise consult the fine manual that came with your equipment or consult a professional with said equipment. This is bog standard.
fucking kidding me.
Time is an illusion. Lunchtime doubly so. ~ Douglas Adams
Set the default gateway to something that only that device uses. Only turn on the default gateway when you want them to access the system ( could be automated by making the default gateway a software service running on an existing machine that you can enable at will ).
There are some firewall/access devices/content filters that restrict access on both time schedules and destination. Maybe talk to your network administrator?
boom goes the dynamite....
Obviously, just put the device behind a firewall. If the firewall operates in bridge mode, it won't use NAT, so the people who insist that their equipment is directly connected to the Internet won't know that it isn't.
The real "Libtards" are the Libertarians!
well hopefully the equipment (and all equipment in the facility aside from the outer firewall) isnt directly connected to the internet.
so then just tell whatever switch/firewall machine that is upstream of the devices not to ever pass packets to/from them that arent
local. and dont give venders access to control your switches/firewalls.
Part of this depends on how they have remote access...is it dial-in? Are they connecting to a jump host via IP connectivity? Is it a VPN? The solution depends on which of those they use, because it's all different. You can use a relay to open/close the actual circuit to the phone line if they dial in; I know a few power companies that use this as a safeguard for their power substations that have dial-up access. If it's a jump host or VPN, then the details of that solution define the approach.
But here's a question for you...what about having a limited time to have remote access would have kept this from happening? From what it sounds like, the process control people would have let them in anyways. And then...what happens if they run out of time, halfway through whatever they're doing? Or even more interestingly, what if they screw everything up (again) but then blame it on being disconnected while they were in the midst of doing something, so they can put the blame on you? This sounds more like a people problem than a technology problem.
For your security, this post has been encrypted with ROT-13, twice.
We usually handle that sort of thing with our Checkpoint firewall (I'm sure other firewalls can do it too but that's what we use). When a vendor needs remote access we put the time limits into the firewall rule. We also make a point of only allowing vendors access from specific IP addresses.
Depending on how your vendors get access you could configure their credentials to only be valid when you want them to be (eg Log on hours in Active Directory)
Apply STRICT policy: breaking the Law is grounds for dismissal (I'm talking Computer Misuse Act here).
It's either walk or face a criminal charge. At the least, a civil suit for any damage caused.
Question: how the holy shitting fuck can you allow a PCS unfiltered access to the Internet?? Are you suicidal?
Operation Guillotine is in effect.
Sounds like a contract/liability issue than a network security issue. (This is what happened to our health system.)
conf t
int gi 0/15 !or whatever
no shut
end
Any switch other than very low-end, consumer grade ones, will have some way to do some variation on the above. Good ones (Cisco/Juniper) will also let you create a timer to shut down the port again after a certain period of time, certain time of day, etc. You could also remotely enable/disable them via SNMP if you want...
You could stick an old machine with 2 NICs running Linux in the middle running a simple proxy written in nodeJS. Since you've written the proxy you can write all the rules you want. With node it would be incredibly easy.
I'm a DCS system admin at an oil refinery. We keep the DCS and business lans totally separated, and that directive is driven from the top down. If anyone asks for remote access we just let them know that's NOT going to happen - end of story! It can be a pain getting files from one network to the other (patches, etc.) but certainly worth the effort.
A modern sslvpn product (or hell ancient IPSec for that matter) can limit specific users and groups by time and application. You can generally get an audit log of what they're doing. For complete monitoring you need a firewall, many solutions allow you to limit access to applications based on user identity. We sell products from both Juniper and Pallo Alto to do this. I'm sure there are others, those are the ones I'm familiar with. We have many customers with process control networks that we protect like this. People can get killed if the wrong machine is started remotely at the wrong time.
I'm posting as a/c so the trolls don't say I have capitalistic motives behind my post.
In this case it would have been a futile effort, since they would have called to have the connection made and messed up the upgrade anyhow! If you could trust the production people to do it, it could be as simple as unplugging an Ethernet cable and only plugging it in when it was absolutely necessary. Other wise a network switch can have individual ports turned off remotely. Not an IT that's my son, but this is what I found on a quick look.
your are describing managers who are incompetent and not qualified to run their business.
you have done what you can do. you arent the company president.
this is like dealing with a durg addict. you cant save them. you cant change them. they have to want to change, and they dont want to.
let them fail. let them go bankrupt.
make sure you cover your ass. keep documentation of their stupidity and your warnings.
and keep your resume updated.
Plug the router's power supply into a cheap timer. Twenty bux, problem solved.
* Carthago Delenda Est *
If this system is using an Ethernet connection, just get a Linux or *BSD box running with bridged Ethernet interfaces or pay for a decent smart switch. Heck, you could probably do it in Windows -- that supports bridged interfaces too.
Simply disable the interface connected to the device you want to protect whenever you don't want outside access. With a Linux/*BSD box, this could be accomplished with simple scripts. You'd probably have to write up a simple manual procedure to do it with a switch or Windows box.
I don't know what type of router you have but many do have scheduling capabilities. Actually publishing information like brand and model of router would be pretty dumb.
My first step would be to contact your router manufacturer and if necessary get one that has that capability. You could even put all of your manufacturing equipment behind one unit on it's own segment of the network with limited access from outside, assuming that you really need network access at all.
Unplugging from the network is an option that will permanently take care of the problem.
Non bene pro toto libertas venditur auro
Assuming you have managed switches a simple crontab entry pointing to a shell script can open a connection to the switch an admin down the port that its plugged into. If you want to get really fancy you can have the outbound traffic going via a transparent squid proxy / iptables so you can tell when the port is in use, and keep logs of the connection state.
You can also go with a non-NAT firewall (bridge mode), which will block incoming connections while the device / people on the inside wont know anything is there.
Honestly a timer on an unmanaged switch isn't a bad solution, it takes any technical skill out of the equation, its (assuming the timer doesn't fail) hack proof, and does not require and maintenance / patching to keep secure.
> As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks
If you're going to call someone inept, you better make sure you're not, especially if its your own FUCKING FIELD.
actually, I didn't need to, I've dealt with several cases involving computer crime. Such activity as the OP describes *may* fall under Section 3.
Operation Guillotine is in effect.
It's the only way to be sure.
The Janitors were always unplugging our equipment, usually at regular intervals. Just place the device in the middle of a cleaning area, with a network cable to short to move it without unplugging it.
Or, you know, you could sue the crap out of your vendor for damages.
How on Earth do YOU get to make fun of other employees at that company? I can think of at least a couple of filtering methods more elegant than a freaking christmas tree timer and I'm not even in IT. If all departments' staff quality is the same as IT I just hope that the "large industrial equipment" is not something that can affect other people.
Filtering access on a per-request basis is one thing, and I see how that's critical and can't think why you haven't implemented this already. Filtering access on a per-timer basis is the WORST WORST WORST idea ever. If I could make that any more caps locked I would. There are SO many things that can go wrong with a blind timer-based disconnection that I won't even bother to list them all, I will just paint the simplest of pictures in a newspaper title: "Incomplete update to a CNC machine leads to hands being sawn off".
Do yourself a favor and change jobs.
I suspect parent is not in the US, so it's not the CFAA. In the UK, it *is* what he called it.
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
Password control? You could implement a policy where all remote connections must be sanctioned by someone on your side and disable/change passwords fatter the remote user notifies you their work is done or after a predetermined amount of time. Bear in mind you should read your support contract before you try this -you may find nasty penalties or be in breach.
I would use an OpenBSD bridge(4). Get a PC with three network ports. Install OpenBSD. Create a transparent bridge between the two networks and use the third connection for access to local ssh(1). I would then configure the pf(4) firewall to allow limited traffic (such as SSH) to cross the bridge. Since the box is a bridge, it's transparent to network traffic and adds an almost negligible latency. Whenever you want to disconnect traffic, log in using ssh (Putty from a Windows box), and turn off the bridge. With an SSH client on a smart phone, you could turn the network off and on from anywhere in the world within a couple of minutes of receiving a phone call. A timeout is easy. Create scripts to enable/disable the bridge and use cron(8), at(1), or a script to fire off the enable/disable scripts at specific times, dates, or intervals.
still doesnt change the fact the guy is a prick.
how do you blowhard about how you want to play lawman and fire people for 'breaking the law' while you have an anarchist quote at the bottom of your comment?
First thing: I don't know how woefully inept the vendors and workers are; it appears that IT doesn't have security controls nor did any auditing to determine liability that existed and then work to actively mitigate risk.
Second: Use Perl. You can create a script, probably two one to run a web GUI (I like PHP for that the GUI) and and another for the systems logic to 'fake' the commands that would shutdown / no shut the ports on your switch. This way, you can enable them from a GUI and use some form of DB to list the unique equipment ID and what switch and port it is on. Then you get a pretty looking, password security focused, ability to be audited via logs, access site that makes turning ports on and off easy for just about any shop jock and at could have a timer built in to shut all ports at a specific time.
Truthfully, I have gotten three raises in my current and past job for doing the stuff I listed above. Anyone can point a finger, its much harder to create the solution that works for IT users at a level that they are willing to interact with.
You do not need to think of any of that if you just stick a managed switch between the internet connection and the equipment.
Enable the port when you want them to have access and disable the port the rest of the time.
Why is this complicated? Why is it a question even?
A Christmas tree light timer ??? How does the OP have a job?
Why is it so hard to only have politicians for a few years, then have them go away?
Either use managed switches to separate the LAN's and lock down the ports to only allow certain mac addresses on the PCS VLAN, or create another LAN with dumb switches. Add a VPN box that is connected to both networks, make it the only allowed method of connecting remotely to the PCS LAN, give the vendor a VPN account, and only enable the account temporarily when it has been approved.
First off this is an issue between your vendor and the controls staff. The vendor would absolutely not update the system on their own as that would easily lead to a lawsuit. Someone OK'd this.
Easiest way...they came in through the business network, so make them require some signoffs to have the connection enabled for a number of days, after that remove access on your end (after checking with them).
As for kind of insinuating that process controls guys are incompetent, that is absolutely not true. This isn't a typical IT system, the servers must be up 100%, the workstations must be up 100%. Sometimes, if production won't give you an outage, you can't make the system changes you want to. Similar to what happened here, you tried to update during a non-outage, and it screwed up, now you are losing production until the system is back online. To be blunt, it is a lot easier to lose your job if you screw up the system.
Believe me when I say we want updated software and completely patched systems, we just want to do it during downtime.
Most Remote Access solutions allow you to specify times you will allow remote access and from what location/IP.
So set your policy rules to only let them in certain hours and from a set location or point of ingress.
Then you can also leave this disabled and only activate it when you want to let them in.
Its never a good idea to allow anyone not directly employed by your organization unfettered around the clock access.
How does the OP have a job?
The real IT guy was fired and replaced with a mid level sales manager. Anonymous submitter for a reason. And maybe they sell Christmas tree light timers and happened to have a few lying around.
“He’s not deformed, he’s just drunk!”
I think the OP is missing something.
I do process control. It's not manufacturing, but that part is irrelevant anyways. The issue at hand is that process control has shifted to control systems that are networked. There are options that don't use ethernet/ethernetIP, but they're increasingly going the way of the Dodo.
We're in a strange time when control systems are increasingly being networked, and the guys that used to do control/automation (and used to do it with relay/hydraulic/pneumatic) don't have the necessary training to integrate the systems correctly. Most IT people don't understand how control systems work and the implications of changing network configurations.
The way forward is to merge IT and process control. Unfortunately, that's easier said than done.
Timer-controlled IP address management can be found in dd-wrt firmware. You can set that machine IP address to have zero minutes and zero seconds of access if you want, and only do it from the router when the vendor calls you to ask to update.
This is literally a $10 ebay WRT-54G with a free firmware upgrade solution.
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
check your contracts before doing any thing you may be on the hook for the full cost of that large industrial equipment after you break the contract
I agree there are too many problems that this could cause (putting the machines on a 'timer') than they would benifit from security. If any company is having mission critical hardware / software handled through the internet you must have engineers or at least senior support staff present to make sure all goes well, after all when are you going to ensure the equipment is running properly, Monday morning 8am?
Have the connection closed with a metal case and a key, when the time comes the company doing the update will contact you and you pay the support staff / engineers to be present. One of the support staff has the key to the lan jack and plugs the cable in, over the phone you cooridinate the service and test the equipment when the work is done. Once all is done disconnect the LAN cable and lock the housing over the jack.
This ensures the right person allows access and is present when the equipment is operated on.
As a bonus you don't find out first shift Monday that production is down.
A loop, by its nature, continues. If that didn't make sense, start reading this sentence again.
You are looking for time-restricted firewall rules.
You can roll them yourself (Linux, Free-BSD), by just having two sets offirewall rules, and switching to the restricted set after the time expires. If you re-inilialize or don't use connection tracking, existing connections get cut. Reloading a firewall rule-set does not cut connections if it does not take too long). You can also isolate this in a specific sub-chain, and then just reload that one to enable or disable the specific connection. That way you have only a low risk of messing up the rest of the firewall configuration.
Better commercial firewalls are offering time-restrictions on rules as well, by basically using the same mechanism.
On the process side, recommend to management that the idiots keeping this access open despite being warned should be fired.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
health Care systems are some times like this in where out side suppliers and vendors have control over systems and some times they don't install updates / say you can't over firewall this system.
...is a post incident review with support people involved, and their management teams, along with directors and executive involvement to identify what the problem was that caused the business to be inoperative for the duration of the incident, what policies and procedures need to be followed going forwards, and so on. Once policies are established, solutions that support those policies can be implemented.
As an example for your situation, since a vendor was involved in an upgrade, that should have been part of a scheduled change. The change should be documented ahead of time as to what is being done, what systems are going to be touched, and who the responsible parties both within the company and external to the company are for that change. Included in the documentation should be the fallback plan for dealing with issues that crop up during and after the change, within an appropriate test window that is included in the change window, as well as clearly defined backout procedures. "fix and fall forward" or equivalent statements are not, and should not be, considered acceptable plans. Wherever possible you want to have documentation attached that the procedures involved have been tested in a suitable test environment. (This may not be possible in situations where a test environment would cost as much to prepare as the production environment.)
As far as limiting remote access, as others have pointed out, such limits are trivial based on what type of remote access is in place, and what policies are established. At the very least account authorizations required for performing changes on production devices should require someone in house approve that authentication, be specific to the time when those changes are scheduled to happen, and should not allow similar access to devices or types of devices not involved in the change.
You never know...
Agreed.
Put in a managed switch, log in the switch to enable / disable the port when you want... <yawn> ...
Examples: 1. Separate all network-connected industrial equipment into a separate logical network (obvious but needs to be said); 2. Either use port security (highly inefficient at scale but usable for one and two-offs), 802.1x NAC, or VMPS to permit physical access to the network based on device MAC address; 3. Gatekeep all remote access through a VPN that permits dynamic access controls to very limited sets of devices (e.g., Cisco ASA but probably Juniper SRX and others).
I am a network engineer for a manufacturing company myself and of course we have many many industrial controllers. The primary industrial engineer who manages and sets up the majority of these came to me just the week swearing up and down the network must be the reason he can't connect to X controller but could connect to Y controller that was identical. The device was pingable but whatever control was being used could not connect and I could not find any open TCP ports (assuming it was using TCP, which the other one was). The remotest possibility was that it was the cabling however the device was quite cleanly pingable so that seemed very unlikely.
Sure enough he is able to climb up high to get at it and it was stuck in a state where it needed to be restarted and bam he could get to it no problem.
99.9% of industrial engineer calls where "it's the network" go like this. I'll admit sometimes I goof on the VPN split tunnel ACLs for remote access to devices that our vendors need and sometimes when I setup new sites I forget to setup the downstream switch VLAN domains properly so that switch won't see the industrial VLANs properly but for long-running networks the goofs are nearly always with the industrial engineers. One time one of the guys put an industrial device on a "user VLAN" and totally screwed the network up because the device appeared to be running some sort of DHCP server. Another manufacturing company I worked at the industrial controls engineers were ALWAYS doing stuff like this.
Anyway, long-term if you really want to get paranoid you may want to consider 802.1x NAC or possibly VMPS (in a Cisco shop). VMPS is a lot less complex and easier to manage but it's a very simple text file database of permitted MACs and which VLANs they should go in. 802.1x is considerably more complex especially as you get into sharing 802.1x ports with IP Phones which of course is extremely common. A lot of NAC servers to manage network access also require agents which frankly is lame. Look at PacketFence for a supported free open source NAC solution.
You are either spoon feeding an idiot or feeding a troll submission...
Even in the smallest shop I work with remote access for maintenance. The login accounts were only enabled for a short time after maintenance had been scheduled and verification phone calls done just before the work was aboot to start.
This ask slashdot reeks of troll baiting an cluelessness if not downright stupidity...
The best solution is to use this event as a jumping point into securing it right... No matter what technical solution you come up with, the weakest link are the people. Education, some firings, and getting a better vendor are the real next step. Remote access can be a marvellous tool to getting problems straightened out without flying people in, but it sounds like these are the kind of people you wouldn't let walk unescorted in the plant...
More Caffeine. NOW
Plug the switch into a web controlled power switch:
http://www.digital-loggers.com/lpc.html
Eight power jacks that can be independently controlled over your network. You can control access to the entire device or individual sockets with multiple users and passwords, and they have built-in scripting functionality that shut off sockets based on the time, power-cycle if a repeating ping test ever fails to get a response, and other options I haven't bothered to look into. A real party. I think they're about $100.
What you want to do is SO frigging simple it's embarrassing. So, what this tells me is that you are not spending money on IT staff. Quit being a cheap bastard and hire at least ONE competent IT person.
Your thin skin doesn't make me a troll
Time based ACLs.
The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days.
This isn't a technology problem.
Through their incompetence, they caused damages. Collect your evidence, hire a lawyer, and make demands. If they refuse to pay, sue them.
Watch how fast they start caring about doing remote upgrades more carefully, competently, and with customer involvement. The only thing companies collectively care about is making money. At the very least, you'll cause their liability insurance rates to go up.
Please help metamoderate.
Cut a network cable and break the wires out, one to each port of one of these:
http://www.sainsmart.com/8-channel-dc-5v-relay-module-for-arduino-pic-arm-dsp-avr-msp430-ttl-logic.html?___store=en&___store=en
Then, connect it to an ATtiny - ATtiny85 should be fine; it's only a couple of bucks - and program it to go on and off as you wish. Side benefit: it runs on batteries in case the power goes off.
If you submit the idea to Make, I get credit.
All Internet connections must cross a Firewall. Disable inbound connections, done.
Seriously, this is a question?
You wouldn't have an osm to worry about.
~Sticky
Which is fucking worthless. I work in Process Control and actually do work all day. IT forces Symantec and shitty Lotus Notes down our throats and then spends the rest of their time trying to troubleshoot the problems...no wait, they make us do that. IT does shit.
A Christmas tree light timer ??? How does the OP have a job?
You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...
I just got done with a contract at a large bank (It's one of the 50 largest companies in the United States)... all their deployments are run off USB drives hung off servers at their retail locations, they have 512kbit backhauls to their corporate locations, run DHCP over the WAN, have no QoS, and I kid you not -- about 5% of the managed switches have been forced to 10mbit half-duplex.
And since they're so security conscious, all the workstations have drives that are encrypted, have antivirus that runs every 4 hours, whether you're using the system or not, a couple other "intrusion detection" apps that also run, sometimes on overlapping schedules, sometimes when trying to patch the operating system... and for the bonus round: An account used for software installation that has full local admin to every workstation... and has a password that's the same as the account name.
-_- Attaching one of those appliance timers to a switch to shut it off at predefined intervals seems so stupidly obvious, but when you realize how stupid the average person is, and then realize that the ones stupider than that work in HR and Accounting, you quickly conclude the same thing the rest of us in this industry have:
Just drink your damn beer and try to drown out the stupid. Thinking about it will only depress you. Trying to do something about it will get you fired. Trust me... there is no faster way to get fired in IT than doing your job well... because you'll get noticed by all the incompetent asshats that HR and Accounting let in, and they'll form an alliance against you to get rid of you. And for the super jaded special bonus round... trying to get shit done will make you realize that the reason you can't get anything done is because everybody has silo'd themselves away with crucial documentation, settings, or knowledge, to assure themselves of continued employment. Start poking around, and they'll feel threatened. When they feel threatened, they'll find some way to go behind your back and make you look bad. Do this enough times and management will consider you an agitator and... ker-chop.
If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.
#fuckbeta #iamslashdot #dicemustdie
That's fine, until the process control guys unplug from your nice managed port, run a cable across the floor and plug into a port that you're not actively managing. And they will do that. If you don't think so then you haven't worked in that type of environment.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
Configure your firewall rules so that access to these systems has to come from the internal network or through a VPN.. Create a VPN account for the vendor but leave it disabled. If there is a legitimate need to access the system they can submit a request through your internal change control process (or have an internal contact submit the request on their behalf) and pending approval the VPN account can be enabled when the work is to begin and disabled again once the work is complete. In fact if you are subject to SOX, HIPPA etc. you are probably required to grant external access this way.
You can probably do the same with firewall rules rather than a VPN but personally I don't like granting external access to internal network resources.
Any insufficiently advanced magic is indistinguishable from technology.
This isn't my field, but I think you should do nothing. IT's job is to provide network access. Process Control's job is to keep the machinery running, and if they fail to do so despite your warnings, it's their ass on the line.
Yes, "not my problem" is a classic way to make a workplace awful, but consider this: if Process Control can't get a software update to their machinery because you've blocked it, and something bad happens (worst-case scenario, a machine kills someone), then it's *your* ass on the line.
By all means give people support in doing their jobs, but don't do their jobs for them.
As someone who's worked for the supplier (not in this case, but certainly other analogous ones), I'd like you to understand a few things.
First, we know downtime is bad. There are only two reasons we remote in to do updates like that on live equipment: One, shit's already hit the fan, or about to hit the fan, and probably not in ways you know about. Two, some jackass is demanding a change before your next maintenance window. (Usually, after only one or two incidents like this, they learn not to do that...)
Second, we're not happy, either, when these situations occur. The software managing your gear is uniquely written to your needs; it's not something we can pull off the shelf, make a couple tiny modifications to and call it good. When this kind of shit goes down, I may well have to put in a few consecutive 18 hour days to get things straight; Sometimes I can't just roll back because someone wanted me to push the change, and if I didn't catch this failure case despite all the testing I'm already doing (despite thinking I've covered all the bases three ways), there's a good chance a retry with an attempted fix will have the same problem.
Third, we're often the first line of defense when something in your system goes sour. When your system starts eating into its safety margins when it's not supposed to, Bad Things are imminent. You don't want to stop me from getting in there and taking a look. You might be right there, but I'm three states away, and if I show up, I'll have half a day of mandatory OSHA training before I can get within 100 yards of the equipment I'm accustomed to viewing remotely. With online access, I can be there in 15 minutes even if it means rolling out of bed and getting on my standby computer. It's the difference between losing 10 million dollars in product or losing a couple orders of magnitude worse in down time, equipment destruction, rebuilding and recalibration.
So that's my story. Do you know the purpose of the update? Do you know the circumstances of maintenance windows?
Why are you asking this? Were you asked to come up with a solution by your company's management?
If not, then you should not be working on this. The main reason is that in a place that has bad politics (sounds like yours), unsolicited solutions are often rejected out of hand, and your rejected solution may be the best solution, but it will never be implemented.
If you were asked to solve this, then you need to begin with getting management's parameters. It sounds like your equipment is connected to the Internet. Is that something that your company's Management requires?
Anyway, where I used to work we had a similar situation for some Internet-connected systems.
The target device has some accounts and passwords for access. The vendor only has access to a single account that is usually an admin account, but not THE admin account.
Whenever the vendor required access, the password for their account was changed and the new password was given to the vendor. When the work was completed, the password was changed again to something unknown to the vendor.
All this went into the maintenance log for those systems that had logs. You do have maintenance logs for this valuable equipment, don't you?
As someone else pointed out, you cannot do this with a timer because it will happen that access will timeout during maintenance and now the device is broken, and that would be your fault.
If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.
amen to that
Agree. They will do this. Seen it everywhere. And if the run is too long for a cable, prepare for wireless.
~Sticky
'cuase I got no real mod points...
Your thin skin doesn't make me a troll
Lots of people have been recommending proxies. Why not put it with the equipment, and control physical access there? If they want to plug it in, they're plugging into your tiny micro computer which is now integrated into the equipment.
That would significantly reduce the likelihood of them pushing an update, but if updates are pulled by the machine itself then the moment it reconnected again it would grab an update from their server
That, or you might find an el cheapo four port switch between a production machine and its normal port where there wasn't one before, so unless you set the max MAC addies to one as a matter of daily business, you may have an ugly surprise.
Scissors
Probably cheaper than a hi-falutin' IT solution; you can explain when to pull the plug, when to put the plug in;
he can do all sorts of useful things around the place; fetch the burgers or donuts; did I say its cheaper? And
in a few years, he can probably run the IT department with a big pay raise, and in turn hire another Mexican
kid.
Ignore the haters, they don't understand the politics for this. I used to design industrial Ethernet networks for a large vendor, and we spent quite a bit of time pointing out to customers how dangerous the direct lines were. However, IT departments have very little say over manufacturing networks. This isn't always a bad thing (see the many IT/help desk horror stories). Because the remote access is often required as part of the maintenance contract, offer to partner with manufacturing to install a small firewall with access filters that are controlled by IT, but set (requested) by manufacturing.
A small Cisco ASA, Juniper SRX or its like will do the job nicely, and can shield you from hack attempts along that access path.
All I am asking is for you to either affirm or deny it: you're a Cheetohs-stained mama's boy who's not been out of the basement since 1997, aren't you?
This is entirely accurate.
Just work for a small company of people who know what they are doing.
This is a problem with the organization of your business and cannot be fixed with a technical solution.
"the main issue is that they were able to do this at all, but reality is that IT gets overridden by the Process Control department in a manufacturing business. They were warned about this"
You need to raise the risks to management and if they choose to override your recommendation then the problem is theirs, not yours.
blindly antisocialist = antisocial
We hear of so many idiots with critical infrastructure connected to the Internet that I felt it my duty to single this post out as #outbreakofcommonsense and say thank you for fighting the good , non-moron fight
Hats off to you sir (or madam as the case may be.)
You said they need remote access and botched the upgrade. There is no technological solution for this. If your management insists an incompetent vendor have such access and thinks you can do something about this result then the solution is to find another job. If you aren't being told to fix it, realize that it's not your problem.
You can't fix stupid; you either have to accept it or escape it.
The I told you so and not our problem method is not what's going to get the right result here. What you've done is passed the blame but not fixed the underlying cause of the problem which is a fundamentally flawed approach to dealing with your systems.
Both the process control team and IT have experience that the other team doesn't have. What is needed is for both teams to sit down and nut out what a site wide acceptable policy for ALL gear is and then have the upper level of management sign off on the policy. This policy should state what is allowed, and what is not allowed, that way management ultimately own the risks.
We have this kind of policy at work. The IT policy comes from the top down and defines not only how the business network is run but also how the process control network is run. There are IT people working in our process control group and there are process control people working in IT to ensure a unified approach.
The fundamental problem here is that the process control guys don't seem to have the security expertise, and the way to fix that is to work together. They should have the facility to allow vendor remote updates. They should have the knowhow to be able to control this link, and the sense and processes to ensure it's not used in an unauthorised way.
If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.
^^ TRUTH ^^
I wish I had known this ten years ago. My soul is like the Cloaca Maxima. Still 4292640 minutes till retirement.
> but reality is that IT gets overridden by the Process Control department in a manufacturing business
It happens in a lot of industries. We're forever chasing vendors who think it's ok to pull our systems out from under us to apply updates, sometimes (thankfully rarely) bricking the systems keeping us down until they can make physical repairs.
I don't think there's a surefire technical solution. We disallow access from outside directly to our hardware via our firewall (the best solution -- don't think christmas tree timer, think firewall or switch controls) but since the outsourcing, our firewall is itself under management from an outside group (albeit a different one) and they don't seem to know what they're doing, except to call an operator to press the reset button when a problem is reported.
But the point is, the problem is a social one, not a technical one. I know you haven't had good results so far, but this needs to be fought in management, not in technology. A major production outage gives you fuel -- get riled up, and go talk to some people. Make it plain that the next time the vendor makes any change at all without first approval from a cross-department board, will be the last act that particular vendor does in your company. Put some teeth in your service contract. Hop to it. Your company is at stake.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
The device is called a "firewall" and is set up by an "IT Professional"
You tell the IT guys when (or if) you want that company to be able to connect in remotely. That's it.
Who is more inept? The inept or the inept who hired himt? ~Obiwan (while working IT)
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
"First things first -- but not necessarily in that order"
-- The Doctor, "Doctor
how about an arduino hooked to a servo that can unplug the hard line after a predetermined delay? also don't forget that the people you deal with are inept at your job because it is your job. not their. Most of the time I see IT folks behave holier than thou and create roadblocks to the actual job. this leads to people finding workarounds and then this happens. and before you dismiss me as those inept ones, I have worked on both sides. just so you know.
-keyboar battt dead
This Ask Slashdot has to be in the top 10 worst Ask Slashdots...
Thanks for the pep talk Danny Downer. Whats a person that loves computers supposed to do then; drive taxis and live off food stamps?
"Is there a device to automatically disconnect network or otherwise time limit a physical connection to a network?"
Yep, it's called a router.
Seems to be something that needs to, at least in part, be governed by a contract with the supplier. You might find yourself the scapegoat if off your own back you buile q Wallace & Gromit inspired steam-powered Kill'O'Network contraption.
-- Using the preview button since 2005
See, the process control guys wants things to work. So either you work with them (putting up nice big signs on the switches with a number to call 24/7 is a good start) or they will work around you.
At the end of the line the process control guys are more important to the production than the IT department is. Sure in a perfect world you want a nice secure IT environment that still functions 100% for the process control guys. But if you have to choose between machines standing still or a possible weakness in the IT network I know what any production manager would do 99% of the time.
/ A process control guy (whom sends a big thank you for all the nice cisco guides on the web that makes it so much esier for us without IT eduction and experience to make your managed switches do what we want them to do once we press the production manager for the login/password/that other password )
ELEGANT! ELEGANT!
Sheesh. IPTables, Cisco timeout functions...no imagination whatsoever.
Write an app for a dedicated iPhone (hooked to a power adapter) that is interfaced with an RJ45 switch (low-voltage control circuit for a small relay in place of the usual rotary switch) for the ethernet cable , so that the vendor has to call that iPhone and maintain a connection to keep the ethernet connection active. Limit the phone number sharing to vendors only.
"Something with two network jacks on it that disconnects the port after a set time"
No wonder some people have a hard time finding jobs, since a lot seem to be taken by cheap labor with unrelated or irrelevant knowledge or workforce repurposed from other departments to cut the costs of hiring someone who has the proper knowledge. I am not in IT, never was, never will, yet even I know of the device the poster seeks with multiple jacks and connection handling functions, which are magic boxes brought by blue fairies in the middle of the night and are called... wait for it... MANAGED SWITCHES!
I mean come on, really?
If you really don't know what to do, then at least run a google query with managed switch session timeout or vpn router session timeout.
I am putting myself to the fullest possible use, which is all I can think that any conscious entity can ever hope to do.
Why do you want to disconnect after a set time interval? Your time will never be correct. Sometimes, legit maintainance might take longer (e.g. if they're monitoring the system to trace a problem) and often it will be much shorter.
Without knowing the details, what you need is a point inbetween that IT Security controls, and the procedure that says enabling remote access requires form 123a filled out. Which would be a simple paper saying "please enable remote access for vendor X on (datetime) until (datetime)".
Everyone will hate you for the added bureaucracy, but this is the one and only way to guarantee that no outside vendor can access your system without your knowledge.
Take care regarding the wording of the form. Nobody likes the beg IT to do anything. It should sound like an order, that'll make them feel a lot better, and you're not planning to deny any of these anyways.
Assorted stuff I do sometimes: Lemuria.org
If you're serious about finding work, try moving to a state that's (mostly) the opposite political persuasion. It's never black and white, so you'll quickly find you have natural allies against the common enemy.
Oh, and quit reading slashdot cold turkey. That's part of your problem.
Make absolutely certain you do not disconnect the connection when someone is remotely working on the system. A lot of industrial control systems brick if you do that and the remote party is doing something more advanced than looking at some sensor values (eg, updating the software). Typically this also breaks the remote management so someone has to go to the physical machine to fix it. Many service contracts also expressly phrohibit you from tampering with the network connection in any way (sometimes even asking for an global routable ip without any firewalling in-between), so check this first. I have even known equipment that simply stops working after some time if you unplug the (at that time dial-up) network connection, since it checks if its licence is paid for every 24 hours (allowing an 72 hour grace period).
Try software development (but not in such a small company that they make you the IT guy on top). It has some of the same problems - you develop some software and for problems connected to that, people will obviously blame you.
But at least you are not the guy who has to try and keep stupidity in general IT use under control. The network intrusion some other guy made possible by mismanaging his equipment is not considered your fault, while IT might get shit for that.
C - the footgun of programming languages
The higher hazard industries are required by OSHA to pay attention to management of change (read software upgrades) as part of their HAZOP (Hazardous operations) process.
I agree with the posters that think that a failed software update is a culture problem. The only way to really solve it is for Process Control, Process Operations, IT Operations, Network Operations, Information security and your safety department to have a common understanding of what your computer systems are, and to what level they need to be secured. If you don't you may find yourself featured on the front page of the Chemical Process Safety Board's website http://www.csb.gov/.
I'm not sure whether it's more stupid to allow remote access to vendors or to let it make updates/upgrades on live systems.
In my poor region we mandate a real test system on which any change needs to be challenged first along with rollback procedure.
Only then we go for the real change.
But we live in a poor region and cannot afford SEOs (Stupidity Executive Officers), only technicians are allowed.
Sent as ripples into the electromagnetic field. No single photon has been harmed in the process.
Here's a thought. If firewall/router configs are too fluffy for the organisation to grasp[1] then use a technology that the end-users are familiar with. For example a router with a power supply controlled by a key switch. Who has the key, when it went out, why do you want it?, how long, what will affect? can be managed by production/shift managers as other things. It also means that there is some bod on site who has to be told by the vendor what's supposed to be happening and gets permission for *that thing* with *certain risks* who is on site and shares responsibility for the tweaking. If the vendor turns up on site then they jack-in to the LAN with their laptop with your firewall controls etc and the LAN connects to the process system via the switched router, and the same chain of *responsibility* applies with the shift-manager carrying the can and so exercising proper supervision.
1- you're incompetent.... what you want to do is networking 101
2- you're an idiot. your job is to help your users, not to piss them off
3- you fail to realize you're having a political/organizational problem, not a technical one
4- the solution you're contemplating is medieval
Honestly, you sound like a gleeful entitled bitch thay just stumbled.on some blackmail material.
The Cloud - because you don't care if your apps and data are up in the air.
If you can't automatic a simple task you shouldn't be in IT. This extends to networking in IT.
1. Managed switches
2. VPN
3. Firewall
I would choose 2 and/or 3 tied to a request/approval process that requires authorization, justification, and duration.
Keep the Classic Slashdot.
Take your homework to stackexchange timmah.
the whole of the problem is with the server end anyway - what's the difference between allowing someone access for an hour, and allowing them access for a day? Or a minute?
The server side either has security controls on it, in which case it doesn't really matter how long someone's connected; or it doesn't have adequate security in which case a 1 minute connection is sufficient to blast a heap of viruses at the server.
Seems to me the poster is just a whiny fool who is inept himself.
If somebody wants remote access to manage "their" system, these are a few of the things you should insist on:
a) Explicit Contract statements describing the methods of access permitted, when it's permitted and by whom.
b) Contract must spell out the type of testing or diagnostic work to be performed by vendor technicians and who on your company side will pair up with them to validate. The buddy system is to be used at all times when the vendor is gerfinkerpoken mit das machinen.
c) Managed Access, only allow them in on incidents of failure for diagnostic work or for upgrades again, pairing somebody inside your org up with them during the access window. No Carte Blanche access at all, no VPN tokens and no dial up. All access must be over a minimum of TLS 1.1 links, TLS 1.2 is preferred.
d) Penalties in the contract for fucking up. Make it so nasty that if the fuck up they'll pay through the nose if they take you down. Sorry it has to be this way.
e) Specify that you require test results and approval of the test results (including how they tested) before any upgrades. Also provide a test infrastructure or subsystem to allow the vendor to deploy to first to verify that what they're saying actually works. Just because they've done testing doesn't necessarily mean it will work with your hardware and your environment.
f) All workers from the vendor doing the work must be based in your country and be primary to the vendor. No third parties are to have access to your infrastructure or systems. Don't let subcontractors who don't know your systems and processes in.
g) During upgrades or problem sessions a hot call is established to let key stakeholders in your company know what's going on and to provide progress updates on rediation or the success/failure of the situation. Keep your management in the loop and make sure they're aware of the scope of any changes.
h) Backout plans must be provided in case of failures of any changes. I realize this may not be possible with some PLC/Process systems but that should also be a consideration when purchasing these kinds of systems.
i) Maintain air gaps at all times between PLC equipment and any network infrastructure that has access to the Internet or Corporate Intranet. No connections to networks with "office" applications or information flow should be allowed anywhere near your process control networks. The exception to this is when the vendor is troubleshooting or upgrading systems or obtaining log data in accordance with the rest of this.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
I have never seen anyone fired for doing his job well. I have seen people fired for being insubordinate and abusive (though not often enough!). Are you sure you know the difference?
Yes, we could use a Christmas tree light timer hooked up to a switch or something like that but I want something more elegant.
http://en.wikipedia.org/wiki/KISS_principle
In looking for something elegant you are going to make it more complex and increase the probability that it will fail.
Graybar has some nice external wiring boxes, very suitable for locking down external connections. I've often recommended similar boxes, the outdoor electrical outlet boxes with covers, for use in small data centers. It helps protect electrical outlets of critical equipment plugged into the wall from being accidentally tripped over or casually unplugged.
Use 802.1x authentication on the switch ports and you can control access anyway you want.
http://www.juniper.net/us/en/local/pdf/whitepapers/2000216-en.pdf
http://www.cisco.com/en/US/docs/switches/lan/catalyst2950/software/release/12.1_9_ea1/configuration/guide/Sw8021x.html
Now I hope and pray that I will But today I am still, just a bill
Or will just connect directly to the device and bypass your network totally.
You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...
While it's true that the larger a corporation gets, inefficiencies and suboptimalities will inevitably turn up, this is simply bad management. Of course, you need *good* management to keep a lid on the problems, and said good management is actually quite scarce.
But the thing is, that "cost centre" vs. "profit centre" is just stupid. It's like asking, which actually makes money for the company? Sales or production? Lose one, either one, and you suddenly have lots of problems.
The only thing you'll find inside a company is cost centres, by definition, because you can only see how profitable the company is by looking at it from the outside.
How I know this? I got a particularly harsh deal, burned out, couldn't get a job afterward, and so spent a while with some written material by a renowned management thinker, one of the few that actually have something worthwhile to say. Ironically recommended to me by the very CEO that watched me burn out doing over two full jobs' worth of IT work.
Anyway, businesses don't run on "IT", they run on "business", so speak "business" if you want to get shit done. Though if the business runs entirely on "buzzwords" instead, just get the hell out. You don't want to burn. They ought to fail eventually, but it's actually very hard to kill a company. Which is why so much of management is bad or worse.
fucking kidding me.
It was approved by timothy, so why are you surprised?
Find someone more qualified to do your job.
GET A FREE $1000 iTUNES GIFT CARD
disconnect a network connection after a set time?
give a kid $10, a watch, and some scissors
http://www.aviosys.com/9258st.html
If you use a controller that supports time limits for an outlet being energized, you can then use that outlet to segment your network. You would do that by having that outlet power a switch or hub that would attach the external network and the internal network.
The advantage of that type of setup is that you have access any time that you need it, but you must actively acquire that access. If you don't trust your supplier with access to the switch, you guarantee that they have access only when you grant it to them.
Programming a solution to a solved problem is overkill.
In this case I believe it's very appropriate. They have a static arrangement, (vendor wants in, someone turns access on, manually) and when they're done, someone's supposed to shut it off. This process has demonstrated a history of being unreliable. So the solution comes down to one of three things. (1) replace or retrain whoever is in charge of the process in the hopes of improving reliability, (2) automate the process that is not being done reliably, or (3) redesign the process so it's more reliable by default.
(1) is often either futile or short-term. Any number of things can go wrong here, immediately, soon, or long down the road. People get replaced, are out sick for a few days, forget, make mistakes, whatever. (3) is usually unnecessarily expensive, or at least difficult and time-consuming.
It's been my experience that (2) is almost always the best solution. I'm a big fan of automation, and "pick the right tool for the job". (where "tool" refers to either evolved monkeys or computer programming) Computers are almost always more reliable than people, never rely on a person to do a job that a computer can do more reliably. Given the OP's description of the problem, a few minutes of bash or crontab to automate the disabling of the remote access is almost certainly the best answer. I do this sort of thing where I work all the time. I get tired of fixing the same problems over and over that people just can't seem to do reliably. I automate it, and the problem disappears, forever. The initial investment of time always pays for itself. Sometimes in a few days, sometimes in a few weeks. Sometimes once or twice over, sometimes a thousandfold.
Sidenote: whenever something around here breaks, I ask myself a lot of questions. Is there a fair chance it will happen again? Could full automation or manually-initiated scripting have prevented it? Should the system have provided better logging before or during the event? Could the system have predicted the failure ahead of time and given us early warning? Could the system have identified and alerted us of the problem after it occurred, before we (or the client...) discovered it ourselves? Could the system have initiated automated damage control when the failure occurred? This is all a part of automation.
I work for the Department of Redundancy Department.
I would hook up an Arduino to this (https://www.sparkfun.com/products/10747) and use it to control power on the internet router. It is not a physical disconnect, but it is probably the next best thing.
You can up the Arduino to listen via Ethernet, look for card swipes, check for physical presence, however you want to control it.
cej102937
It is a general rule that people get the results that they have planned to obtain.
The poster has posted insufficient information to recommend anything of any sort whatsoever.l It is entirely possible that the vendor made it a contract requirement that they be able to access the systems remotely 24 hours per day, and some asshat MBA type went along with it.
If you want to get to the bottom of it you need to have a proper incident investigation to figure out the root cause and STOP jumping to stupid conclusions.
Sorry, your supplier is most likely liable for this.
anyway, we are this kind of supplier, but our system works in a way that we have no access to customer devices unless the customer, in a protected access (aka passwored access) connects TO our home server and requests service. Only then we can break things and blame it on the customer :P
You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...
Umm, IT actually IS a cost center in most companies. My company is a manufacturing firm and I run all our IT operations (among many other duties) and our customers do not pay us a dime for our IT. They pay us to deliver the products we make for them. IT falls in to the necessary/useful expense category. If you are not being paid for the IT then it is by definition a cost center. Lot's of necessary, useful and important things are cost centers - it's not a pejorative. The job of IT is to help the other parts of the business do their job more effectively.
That does not imply that IT is not worth every penny we spend on it. It absolutely is worth every penny and I'd spend more on it if we had the means to do so. It enables us to do things much more efficiently than we would otherwise. If it is well done IT can sometimes be a competitive advantage. Unfortunately a lot of companies don't do a very good job with it.
Holy fucking shit. Pot, meet kettle!
Problem solved in two easy steps: IT sets up and manages a Firewall between the Corporate Network and the PCS Network (reason: IT run the rest of the network that the RA/PI traffic travels over) and provides remote connectivity via the Corporate WAN. PCS looks after their shit, IT looks after theirs. Vendors want remote access, business wants stats via PI or similar, IT punches the relevant holes in the firewall to make it happen, and (outside of this) never the two networks shall meet.
A couple of other observations: :-)
- PCS vendors generally lock down their systems to the Nth degree. If you're getting PCs from your IT folks to run your PCS Operator consoles on, then you're doing it wrong from the get-go. There's a reason why PCS vendors provide their own (*cough* Dell *cough*) workstations preloaded with Windows and their software.
- I've blocked more viruses coming from a PCS Network heading to the IT Network than I have going the other way. (Point of interest: If I had a nickel for each time I've nuked a virus that's been traced back to a field service tech who jacked in their USB to an Operator station thinking "She'll be right"...)
- 98% of conflicts are generally caused by lack of defined boundaries between the two organisational groups. Again - PCS People look after the PCS, IT People look after the IT Kit *and* the Firewall in-between. The remaining 2% of issues are generally remedied with quick application of a LART to the relevant parties.
For all the bellyaching about "IT needing to work WITH the PCS guys"... it's a two way street. PCS also needs to work with IT, rather than packing the holier-than-thou-juris-dicktion-crap that they appear so fond of parroting.
TL;DR = Can't we all just get along? :-)
Dude, you and I need to hang out.
Just about every SOHO and enterprise router has had a time based ACL option for the last 10 years.
Even my cheapo 10 year old linksys router has this feature.
Simply match the remote access ports/protocols you want to block and don't allow them during the time range.
But honestly, if IT management was that incompetent, I would be looking for another job.
And they will do that. If you don't think so then you haven't worked in that type of environment.
I see you have. :)
So, this gets to the heart of the matter. The OP needs a meatspace policy, not [just] an ACL. That policy needs to specify consequences for routing around company security. The best time to get this through is before the critical system comes back online.
If the management is unwilling to give him that policy, then he needs to let them know what will happen, and preferably get a paper trail ACK on that. If management still wants to make such outages his problem, then he needs to find a non-abusive employer.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Yes, it's called "a firewall" They do many very useful things. Allow them access on a port, and have a script running that disables it after a time.
Lock MAC addresses to a single port and/or firewall everything that IT can't maintain control over.
I am becoming gerund, destroyer of verbs.
The latest seem to be terminal servers (e.g. Oracle Secure Global Desktop) with time managed access to certain services.
Vendor calls customer, customer grants access for a specific user and system for a limited time, all on screen actions are recorded.
In the past more funny things existed. Some vendor support was by modem dial in only (e.g. EMC), so the customers had a switch that connected/disconnected the physical phone lines to prevent unwanted access.
Configure your VPN headend to authenticate against a RADIUS host that is configured for a one-time-password. You must provide that one-time-password to the vendor each time they wish to connect. The second time the same password is used, it should be denied. This should NOT be a token-derived password they posses, but rather something they must get from you over the phone after authenticating themselves in some other way.
Ensure that the connection has a timeout of some reasonable time that won't kick them out of a legitimate activity.
$ man woman *
-bash:
It's just a troll....
Seriously?
I work in IT at a trillion dollar financial institution and have seen quite a few of the things you mentioned - and worse. Nothing surprises me any more.
And as for IT shitting in one's soul? Yeah, it did the past two nights - I got called repeatedly by another team that fucked up, started interrogating me, and clamored for me to "just figure it out." Who said on call was just for break-fix - I feel like I'm someone's bitch on a leash.
The crazy part: I still love computers. But, for the love of $deity, I hope I finish my master's in computer engineering soon so I can finally GTFO and get a real job.
There's nothing wrong with the remote company.
And there's nothing wrong with the local sys admin
There may be a problem with your senior directors. They don't seem to have a Security Policy
If they have, there may be a problem with that Security Policy. It may not have a 'third-party control' section in it, and a paragraph which says that remote third party access shall only be allowed under control.
If there is, there's probably a problem with your legal procurement team. Because they haven't written into the contract that third party access will only be allowed when specifically requested, that there will be proper logging and roll-back procedures, and that everything must be cleared by local admin...
There. I think that just about covers it...
I think the word you are looking for is "reason", as in "The reason?".
Why are Americans so incredibly STUPID nowadays, and why do they just insert a completely random 'close enough' word because they are too stupid to remember the correct word? Why do Americans keep putting 'then' instead of 'than', or 'that' instead of 'than'? How stupid do you have to be to NOT understand what those three words mean?
Hahaha.
Yeah f-ing right.
Get a $1600 plane ticket to nowhere, overbooked, because you want it now. now now now.
Drive onsite to your craphole kingdom out in the corn fields.
Walk to the boring front desk and wait. Pissed off whoever is like "why didn't you call about this visit ahead of time"
(even though I did, 7 out of 8 people know I'm going to be there... you just happen to be the 8th.)
Watch your shitty safety videos. Listen to "how troublesome machine is" speech.
Sign off on your non-disclosure sheets for your "top-secret" process that everyone else in the industry is doing.
Wait for your machine operators to stand around until their shift ends, meanwhile the machine is doing nothing. I'm doing nothing.
Can I make the chang-- nope, were' in production! But you're not doing anything! Nope, not enough time to run new job...
It's all about the feelings of control in your otherwise pathetic life, to make me stand there for hours as "punishment". Sick bastards.
Because we failed to predict the "mission critical" thing that was never mentioned or tested. Even if asked as part of the sale (what does it need to do).
(Can you provide test product? Nope, costs too much to ship. oh by the way we need the machine 3 weeks early. and give us a discount.)
(lots of time to think about this disaster of a project).
Finally at about midnight connect to the machine and download a change that takes 3 minutes, from a laptop that they won't even give an extension cord to power.
Be like... okay, where's the test product? hello? skeleton staff just smart enough to turn off the lights and air compressor.
Pack up and leave.
Phone call at 5 a.m.
Whining about how the update (that impossibly) changed this or that. Update was some very specific very narrow change. Just some stupid misaligned sensor or setup error it's assumed.
Race back in at about 6:30 am because nearest hotel is godawful far from plant.
Constant phone calls of "we are down". (burn in hell for leaving 6 messages for somebody already coming to your place).
Oh look, you forgot to turn air on to the machine. and look, here's the message saying you forgot to turn air onto the machine.
Questions about why your update "broke" the machine.
"You know how much this downtime costed the company???"
It's hell to be a machine manufacturer.
The pay isn't even that great.
And IT is just another obstacle, a gatekeeper and an entity of "NO!" instead of trying to be part of the solution.
Technical people always try to find technical solution for the problem, where on the other company technical people try to understand what the he*l has been done and try to fix with different way. Solution for your problem is: Talk with all the groups in all companies affected and make list of rules which you all will follow. Technical problems solve easily that way and they are not technical anymore. Make sure that you name responsible person for each company.
You could try placing a cheap router with DD-WRT or some other custom command software embedded, and script it to auto-throttle down to 56k or below if certain events occur (i.e. inbound connection attempting to address the equipment). This gives you the control you need while at the same time possibly giving you a small measure of deniability, since a crappy-looking internet connection will raise fewer eyebrows than a connection that drops out entirely, and it establishes a legitimate reason for you to require the vendor to call ahead before attempting anything like this again ("oh, sorry, our network service has been a little unpredictable at this site, but if you call ahead we'll try to clear some of the other traffic and give you a clear line" or any other suitable lies).
But, as many other posters have said, your first order of business is to ask your company's lawyers if they intend to take legal action against the vendor in question, and also if any attempt to disrupt network access could have legal ramifications (i.e. contract breach with vendor, OSHA liability if the equipment you have seized control of kills a worker, etc). While you are right to want to think of ways to prevent this from happening again, you also need to be willing to let the guilty parties accept responsibility for the problem and propose a solution before you decide to become batman and swoop in to save people from their own stupidity.
Wouldn't almost any managed switch with a command line interface or SNMP do this? You could remotely and on a scheduled basis tell the switch to enable or disable traffic to the port that your equipment is connected to.
No ethernet, no internet, no remote access.
Article X: The powers not delegated... by the Constitution...are reserved...to the people
There are two faults here. One is yours in that you left your equipment connected to the network such that external access was allowed. The other is the supplier/vendor of the equipment who performed a modification of the equipment without your explicit approval. IMO, the supplier is fully responsible/liable for ALL of your costs and/or losses caused by this action on their part. I think an attorney would agree, unless there is a clause in your support or purchase/license agreements that allows for this, in which case, caveat emptor! As for a network connection timeout, this is not really feasible. However, proper configuration of your network firewalls (I assume you have such?) should mitigate this sort of unauthorized access. If the equipment doesn't need local network access for management/monitoring purposes, then simply disconnect it. If it does, then the firewall rules have to be adequate to block remote access without your permission and intervention.
Sometimes, real fast is almost as good as real-time.
Quite the rant. Having been one of the fucktard IT drones for a decade, I can tell you that the feeling was probably mutual. It's more likely than not that the IT guys you worked with didn't have a lot of choice about what patches got pushed when, especially when production guys claim to be too busy/important to create a test environment to roll out the patches to first. On the other hand, there is also no shortage incompetent IT guys.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
Ok - Let's be frank about it. You have a Configuration management problem, not a technical problem.
This outage is costing dollars or pounds.... lots of them. You have no effective configuration management, therefore you have no way to control your environment and no way to control the feckless morons from breaking your shit.
Let me put it this way - do you have a Disaster Response Plan? Do you have copies of scripts and backups of servers held off-site? Do you hold the configuration for the PLCs as a backup? If the answer is "no" to any of this, you need to get the stakeholders in a room and bash the fuck out of them until you get a CM process going.
From there, you can build-in your architectural changes, process control management, DRPs and the like.
If the company wants to have remote access for equipment upgrades, these need to be planned and the risks accepted prior to starting.
If you want a method of kick-starting this conversation - start talking money. How much did this outage cost? $200,000? It's easy to work out.
A Christmas tree light timer ??? How does the OP have a job?
We use exactly this solution for the public WiFi outside our cafeteria, it was an almost free solution and it keeps anyone from doing something like downloading porn or hacking someone from that connection outside of the few hours a day it should be enabled when there are people around. Newer routers can enable and disable SSID access on a schedule but we were working with a unit without such a feature, plus the timer is more green =)
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
ITT a guy asks /. how to perform basic elements of his job and fix his corporate structure all in one!
www.AddaBazz.com It Is New Social Site in World
All in One Free Software Site, www.ansaribd.blogspot.com
... a necessary expense you should spend as little as possible on. Unfortunately, that's what they teach MBAs and the like in college.
You are making a biased assumption not based in any actual fact. I assure you that is not taught in any MBA curriculum from first hand experience. I have an engineering degree but I also have a business degree. I'm an engineer but I also am a certified accountant. They do NOT teach that IT is the same thing as paper towels in business school. I know because I've been through the classes myself. In fact they actually spend a surprising amount of effort trying to teach how IT can be a productivity multiplier. You would be surprised how many people who go to study business management are from IT. Probably 10-15% of my class had some form of IT background. They were there to learn how to more effectively manage which involves more than just optimizing IT.
Now there are managers who don't really grok IT and do regard it as an extravagance. Sounds like you've run into some of these fools. My sympathies for that. On the other hand there are plenty of IT people who greatly overestimate the importance of what they do and think that money should be poured endlessly into IT without any concept of the effect on the rest of the business. That door swings both ways. The trick is to find that nice middle ground where IT can be a real asset without costing the company needlessly.
Create an adapter that physically transforms RJ-45 to some obscure connector format, then epoxy that into the ethernet ports on the devices. Then you keep the adapter. Anytime someone wants to connect the device to the network, they have to come track down the adapter from you.
Realistically, just produce documentation showing you warned IT that this would create a shitstorm and take it to the big boss to show how IT dropped the ball. The problem should solve itself at that point.
I'd suggest getting on the same page with Management about how to prevent this from happening again. Setting times when they can connect may harm your SLA. Don't pretend to be an expert or force your opinions on anybody just calmly explain what happened and what you think the group plan should be to prevent this from happening again. Once you have a plan your company has signed off on, then communicate with the vendor. Don't vilify them. Just say when something like that happens, nobody looks good and you want to work with them to get a plan in place where everybody is protected. Work on improving communication by requesting notification before they connect. To keep them honest do the following: get automatically notified when they connect get automatically notified when they disconnect Set a timeout after an hour If they don't call you, call them so they know you're watching. The second issue is having the internal ability to restore the system when something like you mentioned happened. You may end up owning the whole situation and may have to do learn new things and take on new responsibilities. Work may pay for some training or you might have to Google stuff on your own time and work on an ASA you got off of eBay. In either case, it's a great opportunity to learn and remind management why you're there. Mistakes are an opportunity to improve things. Stay focused on the results and don't get bogged down by the people who aren't concerned with improving things. Change isn't always easy and egos often get in the way.
If you want to block the manufacturer from access to your machines, put them behind a firewall (I like Linux + iptables, because it's cheap), then put the right filters so they can't bloody get to the machine.
You do /not/ want a christmas tree timer providing access for N minutes, etc. Because in the digital age, it likely takes much less than N to trash your equipment.
-Ken
...and solve it with hardware. The tricky bit is getting the vendor to put their fingers on the anvil right before you smack them in the knuckles with a ball peen hammer.
Now, make sure that you can connect to your department's firewall from the outside world.
Now, reconfigure the firewall so that it blocks all access except to/ from your office, and make that the machine's default route.
If the vendor wants to do something to the machine, they can do it from your office.
You may need to fire the first person to disconnect the machine from the firewall, and publicise the fact ; you may need to hide the firewall in a locked IT cabinet ; you may need to monitor the state of the physical connections. These are details that a competent IT person should be used to dealing with.
When the vendor wants to fuck with the machine again, you do your QC on their procedures before letting them even think about accessing the real machine.
Personally, I'd use the super glue and side cutters.
Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"
This is not a technical problem, it's a political one. I know this doesn't actually answer your question, but this needs to be said.
In your own post, you told us the answer to the problem: leave equipment disconnected unless absolutely necessary. What you want is a solution to bypass best practices without doing things completely the wrong way. You're really asking for a lock that will keep a door half-open without letting it open completely. (What's to stop the vendor from applying a crappy update during a time-limited session? What happens if the update process gets cut off half-way though? Now your equipment is completely bricked and they can blame you for tampering with their connection! At the very least, it will create a shit storm for your legal department; at the most, it will leave your company footing the bill to fix the system.)
I also think that you haven't considered that this solution probably won't work anyway. You have another department ACTIVELY SUBVERTING YOU! Do you think that the process folks are suddenly going to back off when you start killing their network connections? Maybe you could play dumb for awhile, but once someone figures out what you're doing (or convinces a higher-up to force you to 'fix the problem') it's going to blow up in your face. If you think that you have political problems now then just wait until your trickery comes to light. In any case, they're going to demand that you give them "time limited" sessions that run for 24 hours or even longer. Also, they might just use their political clout to force you to remove any restrictions that you put in place anyway.
The ideal solution (which you're already tried) is to educate the process department on the best practices and encourage them to implement them. However, if that's not going to work then you need to resort to plan B. Inform them of the risks in no uncertain terms (this also seems to be something that you've done). Do this often. Send them memos, send them emails, etc. and save their responses. If they send you an email telling you that they're not going to secure their network, then save that email. (You work in IT, so I don't have to tell you to keep multiple copies of this email in many different places.) This way, when shit hits the fan, you can point to all the warnings you gave and thus save your ass in the ensuing investigation.
-1 disagree is not a modifier for a reason. -1 troll, flaimbait, redundant, overrated are NOT acceptable substitutes.
In the Linux world we call it scripting the iptables. What you want to do is rather easy.
The constraints of a REAL productin environment can be many. It's easy for you "real IT guys" to say he shouldn't have a job, but you've no clue about the constraints he operates under. I have an active MD and an active OpD above me in my company and regularly have to fight to avoid having our network integrity compromised by simiarly stupid decisions including; 1. Allowing a customer direct access to our SQL server - access to views and/or via web services wasn't acceptable to them. Problem here was our MD is talking to senior procurement managers in the customer company. Thankfully I have a personal friendship with the customer company's IT guy, so when I phoned him we worked it out that a simple web service would suffice; 2. Allowing a vendor to connect critical equipment directly to the net bypassing the firewall 3. Allowing a vendor root access to our domain The truth is I don't always win the argument, so my environment also had a few stupid and dangerous elements. Another truth is I'd never make the mistake this guy did to ask for help on here. Most of you "real IT guys" haven't got a clue what it can be like in a real production environment, so you just dispense smart ass sarcasm with no other purpose than to inflate your own egos at the expense of the poor guy who, lets face it, must already be under more than enough pressure and humiliation without your contributions
So, so true. I was a consultant for several years, then shit happened that was really all I needed for an out. I still love computers but there's no fucking way I'm going back into the industry,
Operation Guillotine is in effect.
it's like going into business with your best friend: bad fucking news.
Operation Guillotine is in effect.
If you can just flat out block the ports that the connection uses, and only open them if you have a scope of work / change request in hand.
Not really sure what other options there are.
Whoever allowed that is a freakin' moron.
If you're allowing a vendor to update your production stuff whenever they feel like it and not coordinating with you, then whoever decided on that within your company is a bloody idiot who deserves this.
I feel your pain, but you (in addition to making sure this doesn't happen again) should be making damned sure to scream loudly that you told them so.
If you blindly trusted your vendors and game them that kind of access, something has gone horribly wrong at your company.
This sounds like a case of being lazy and stupid instead of really thinking about your business needs.
Lost at C:>. Found at C.
You do not say what your place is in the organization. This will directly affect what the proper answer is to your query. From the hip, it sounds like you are some IT minion and that you are attempting to vastly overstep your authority.
If you are management and have the authority to make such a decision, you merely tell the network folks to set up a rule for the firewall or router to simply disallow access except at specified times. This rule could be automated easily but your network folks will know how to do that.
If you are not management, you do whatever management tells you to do. You could tell them, "I told you so", if you had warned them about this possibility previously but that lacks political/social grace. Ideally speaking, you would bring up the possibility of restricting access for the vendor to management. If you are an IT minion asking how to implement this, you are in the wrong business.
Regardless, everything that happens, occurs because management set it up that way. It is not your place to steer the business. At best, it is your place to make recommendations... but even then, management is not required to follow your recommendations. If management is doing stupid things that you know will make the company crash, your only recourse is to start looking for another job before your current one goes away due to the company failing. Correct and incorrect is defined purely by management.
Good luck.
"Someone needs to talk to the tree of liberty about its ghoulish drinking problem." by ohnocitizen