Slashdot Mirror
Ask Slashdot: Patch Management For Offline Customer Systems?
New submitter Nillerz writes: What, in your experience, is generally the best way to distribute patches in a way so customers can download them, considering that the machines are offline? Are there any software packages (open source preferred) that pretty much allow engineers to upload a patch with a description to a web server, and allow customers with credentials that are registered in LDAP to browse and download them quickly? And if not, how do you distribute patches to air-gapped machines?
Assuming that the machine will never, ever be exposed to the Internet, and that physical security ensures arbitrary code will never run on the device, why would you ever patch it?
To fix non-security related bugs.
How exactly would they download patches from a web server if they are offline?
Stand up 2 WSUS servers, one on a connected network, one on the disconnected network. Download all products and classifications that you need on the connected one, sneakernet them over to the disconnected one and import them. https://technet.microsoft.com/... If you're going to need to patch custom applications, then you're going to need to look into something like Configuration Manager with SCUP if you want to have compliance data for the installation, otherwise you can just create a software distribution package.
Ship encrypted files on flash with instructions for them to call when the media arrives. Provide phone support to walk them through the install process, where you provide the password to the files at that time. Once the patch is installed, walk them through formatting the flash media and mailing it back to you.
If you really want to be fancy, make the installer check for something that is supposed to be on a legitimate customer system before it even prompts for credentials to decrypt the files, to make sure that it is being used on the correct machines and that it actually is the customer calling.
Do not look into laser with remaining eye.
And if not, how do you distribute patches to air-gapped machines?
Sneakernet + authorized tech + calendar/ticket tracking
Silence is a state of mime.
Or maybe you might have an airgapped "kiosk", with a keyboard and/or mouse and a dedicated application running modal (so it can't be bypassed to access the OS, perhaps without some hardware hacking). If it's non-networked, or only networked locally to some other system on-site, but still accessible to "users" who aren't fully trusted to the same level as the CEO (e.g., line employees, general public customers, etc.), you might want to patch it *for* security vulnerabilities, such as "if the user presses Ctrl+Alt+Del, they can access the desktop" (or something equally based on the concept of user input -> system access). That would be an example of a software-based security exploit on airgapped equipment.
...that a randomly generated nonsense question just slipped by Ask Slashdot's editorial board.
Am I really getting so old that people find this to be a legitimate question?
Have I really been doing this for so long that there are now people who don't remember a time when disconnected machines was the norm?
Am I the only one left here who remembers (for example) dialling for hours to get into the local IBM-run BBS to download the 21 diskette images needed to update OS/2 2.1 to the latest patch level, digging into the cabinet for several boxes of diskettes, de-imaging each and every one of those diskettes (my machine had two 3.5" floppies -- I could manage two at once!), rebooting off the first disk, and then feeding diskettes into the machine one at a time as prompted to update it?
Honestly -- this is a long solved problem. Some of the technologies have changed (you probably don't need 21 USB thumb drives to contain your patch), but the basic idea remains: provide a downloadable patch image suitable for your application, have customers download it to a USB drive of sufficient size, and then have them boot from the drive or run a script or application from the drive to apply the patches.
And if you're in some industry where you worry about your patches getting out into the wild or need to ensure patch security/validity (where hashes aren't good enough) or something odd like that, put your images onto USB thumb drives yourself and ship them to your customers physically (encrypted, if you have some reason to be uber-paranoid).
Now get off my lawn!
Yaz
I worked on an airplane-based system, and we had removable hard drives which we swapped any time we had to update the software. This way, each upgrade also restored the system to a pristine condition.
I've also done this with CD-ROMs. One nice thing about booting and running from a CD-ROM is that it's impossible for it to be "hacked" (short of creating a new version and sneaking it in to the physical machine).
What companies do is set who is responsible to update such PC/server whatever. Notification is send via email,software is available for download on company website. Trained engineer needs to download it, transfer, install.
I also know that they don't do it for a year or two but that is other story.
...but still accessible to "users" who aren't fully trusted to the same level as the CEO (e.g., line employees, general public customers, etc.), you might want to patch it *for* security vulnerabilities...
It amazes me that CEOs are perceived by anyone to be anointed with some sort of perfect trustworthiness or for that matter ANY level of trustworthiness higher than any other employee.
For Windows machines, I use WSUS Offline. It also comes in handy when I'm at a customer site and their internet is so slow that I can't patch a single machine in a day. Yes, there are still areas of the world where DSL is sold less than a Mbps download.
ASCII tastes bad dude.
Binary it is then.
Microsoft has a product called Microsoft Baseline Security Analyzer, when you combine it with the WSUS CAB file, it will output an XML file of all patches installed and (more importantly) not installed on your machine.
With some small scripting (VBS, Powershell, etc), you parse the XML and find the needed patches in a patch repository.
Then you can remotely push all of that out via PS-Remoting or PSExec, and your offline/air-gapped network can stay patched.
Maybe it was a typo. - "users" who _ARE_ fully trusted to the same level as the CEO (e.g., line employees, general public customers, etc.). Because around here we assume the CEO is no more computer aware than the guy guarding the loading dock.
None of them can see the clouds; The polished wings don't care.
I feel like I must be missing something important...
Why can't you just put it on a cd and give them the cd?
...but still accessible to "users" who aren't fully trusted to the same level as the CEO (e.g., line employees, general public customers, etc.), you might want to patch it *for* security vulnerabilities...
It amazes me that CEOs are perceived by anyone to be anointed with some sort of perfect trustworthiness or for that matter ANY level of trustworthiness higher than any other employee.
The person to whom you are replying obviously is a corporatist Amerikan with an unholy alliance with Satan...I mean Wall Street Bankers.
It somehow jumped the air gap and ended up in the industrial controllers in the Qum nuclear facility in Iran. It will find a way to get to your customers.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Hate to plug a windows product.. but we use shavlik protect with cloud.
It allows you to control what is patched and handles laptop users who don't always connect via VPN often.
The product needs work to work in larger environments. GUI is sometimes a little confusing too. Things are not where you would expect them for easy use.
It also patches our ESXi environment.
Easy to patch, robust, secure and scalable. Just do it!
You have to have someone upload the patches to the airgapped machines using the sneakernet.
I'm not quite getting what we're talking about here? Operating system patches? or program patches?
Program patches are frequently offered in an offline format and MS offers patches like that as well.
So I don't get what we're talking about here. What are we patching?
To paraphrase Samuel L... Be specific, motherfucker.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
If you are patching at the code level, I believe that git is a very good option. Because the git repository can be moved as a standard directory and essentially contains all the patches and history. So even if the customer as custom patches, they can probably easily rebuild around it.
And since a git repository is essentially a directory, you can simply put it on a disk or flash drive and send it by snail mail or courier.
Wsus. Easy enough.
git bundle create <file> <git-rev-list-args>
09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
5.x supports 'disconnected' mode, and 6.1 will support it.
For Debian and derivatives, there is apt-offline. Allows you to write current patch state on an unattached machine to external media, and then get new patches and download them from an Internet connected machine. It can share a common cache dir on the removable media for multiple machines, so you don't have to download the same patch more than once.
or email the patch if small enough
You're OFFLINE, bust out the thumb drive, image it, and start making your rounds to ensure they are properly applied like a good IT guy.
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
I'm not sure what else your problem is. Give them a patch file. What the fuck?
tarball
No, the CEO will be even less computer aware than that guy. Most likely, the CEO will have his secretary print out his email. And read it to him.
Sleep your way to a whiter smile...date a dentist!
> What [...] is generally the best way to distribute patches
Patches for what? For PC operating systems? PC software? Embeded computers?
> in a way so customers can download them, considering
> that the machines are offline?
Well you can't download anything when you are offline. You mean that customers download the patches, put it on removable media and install them on their machines...
> Are there any software packages (open source preferred) that pretty much
> allow engineers to upload a patch with a description to a web server, and allow
> customers with credentials that are registered in LDAP to browse and download
> them quickly?
Yeah like SFTP server for uploading and web server (f.e. Apache with LDAP modules) for customers?
What exactly are you asking?
when the drive starts up have a message come up "Good[Morning|Afternoon|Evening] Mr %name% Your mission if you decide to accept it is to patch this system with the included files. As always if you or anyone on your team ..." and then when the patching program completes
"This message will self destruct in 5 seconds"
this will prevent reuse of the media
I've only had to deal with air-gapped machines on one project, but we used to send out engineers/developers to upgrade those machines. They were security-critical, so we couldn't have customer support staff getting root access to do upgrades, and they were too operationally-critical to trust to automated update disks (not USB sticks -- those can be modified in shipping.)
So we sent staff to do the installs and updates. Not that there was a huge client base, and it was a pricey project, so the cost was negligable. But the customers were not willing to have anyone other than a trained technician doing the upgrades who was ready to deal with any "situations" that might arise. And they did arise -- twice.
Had we not had technicians on site during those issues, the customers would have been down for at least a day while staff were flown out, and we would have been lynched.
I do not fail; I succeed at finding out what does not work.
We see our customers using a few technologies routinely to update "air gapped" networks, to apply updates from standard sources: Anti-virus vendors, Microsft, etc. - no special software is used on the patch/update repository:
(1) If safety is the goal (eg: power plant control networks, train control systems) people deploy a Waterfall FLIP. The FLIP unidirectionally replicates industrial servers to external networks so corporate users can see the data, and reverses direction on a schedule to pull updates. When oriented out of the network, the FLIP hardware is physically unable to send any signal or attack back into the control system network. The orientation reverses on a schedule, typically only briefly because we don't want to let the corporate replicas fall too far behind the industrial sources. When oriented back into the protected network, the FLIP software reaches out and pulls updates from AV, Microsoft, Linux and other vendors periodically, the software checks crypto / signatures as configured, does virus scans, and pushes good updates into the protected network. The FLIP software on the protected/inside/receiving network repeats the checks and sends clean updates to a WSUS, AV server or other repository on the control system network.
(2) If confidentiality is the goal (eg: a classified network), deploy a Unidirectional Security Gateway oriented into the protected network. The gateway software automatically pulls updates as above and sends them through the gateway hardware into the protected network. Nothing ever gets out - the gateway hardware prevents any signal or message from ever reaching the external network / Internet.
We see a lot of people doing manual updates as well. A Unidirectional Gateway replicates servers from a safety-critical or reliability-critical control system network out into corporate. When updates of the control system are needed, those updates are pulled manually from whatever website and crypto signatures and other authentications are checked manually on a corporate workstation, and approved updates are written to removable media. The most cautious customers use CD-ROM instead of USB, because of the CPUs and hackable firmware embedded in all USB gear. They carry the media to a workstation on an isolated "cleansing" network. They scan it again with anti-virus, and again check crypto signatures & hashes. They burn a copy to brand new media, throwing the old one away. They carry the new CD/media into a dedicated control system test network. They install and test the update. When it passes test, they carry the update to the live control system network.