Slashdot Mirror
Ask Slashdot: How Do You Handle Hardware That Never Gets Software Updates? (hpe.com)
New submitter pgralla writes from a report via HPE: Many devices, designed for both long-term and short-term use, were shortsighted when it came to flexibility. How do you handle the hardware that never gets software updates, such as embedded systems and task-dedicated equipment? The article that pgralla shared provides the example of medical devices running Windows 7. "Many of the current generation, when they were first released, used Windows 7, and the devices still work well enough that they remain in service today," reports HPE. "But Microsoft ended mainstream support for Windows 7 back in January 2015, so the operating system gets updated only with an occasional security patch as part of Microsoft's extended support. In January 2020, that extended support will end as well." Many IoT devices are in a similar boat as they're powered by embedded Linux and are not designed to be updated after they enter service."
Of course, these outdated devices create all sorts of security concerns. "Hackers and their access to knowledge and computing power only go up as the years pass, which means that long-lived, fixed-firmware devices become ever more insecure over time," says Michael Barr, founder of the Barr Group, which provides engineering and consulting services for the embedded systems industry. The WannaCry ransomware hack in 2017 affected not just PCs but also medical devices, and ended up costing businesses $4 billion.
Of course, these outdated devices create all sorts of security concerns. "Hackers and their access to knowledge and computing power only go up as the years pass, which means that long-lived, fixed-firmware devices become ever more insecure over time," says Michael Barr, founder of the Barr Group, which provides engineering and consulting services for the embedded systems industry. The WannaCry ransomware hack in 2017 affected not just PCs but also medical devices, and ended up costing businesses $4 billion.
With a single shot of whiskey.
-Clutch Nixon
....don't buy it.
I've seen SO many people whining about MS' forced reboots, etc. STOP!
If there is not a sensible option available, demand that your vendor make a version that can be sensibly updated. Too many purchasing decisions just don't have any sensible criteria. ("Oh, it's built on Win XP and you aren't updating it? OK - scratch!")
Many old tools are computer based
Some old CNC machines run on MS-DOS and a 286 processor
As long as the hardware stays alive, they continue to do the job
If they must be networked, restrict their access to the local net
I have a number of Rohde and Schwarz FSEB and FSEA spectrum analyzers. These cost at least $80,000 new (I bought them used for a few thousand at most). They come with an old version of windows. I similarly have other electronic test equipment with old Windows or even old Linux which the manufacturer doesn't update any longer. For the Linux-based ones I could hack in a new Linux and make it use the old ABI, forget about Windows.
But what really clued me in was that the Rohde and Schwarz equipment had a battery soldered on the CPU board, and it was an hour-and-a-half service to get to it. A lot of stuff had to be removed.
Similarly, my Tektronix 500-series oscilloscopes had two 40-pin DIP Dallas Semiconductor battery-backed memory and clock chips. The batteries in these die and they aren't socketed. When the batteries die, the 'scopes lose their calibration. The company won't give you the program to recalibrate them.
The manufacturers just want you to buy new ones.
So, obviously I back SDR-based test equipment that's Open Source. Who needs a company that wants to screw you?
Bruce Perens.
Medical devices with Windows 7? That's a laugh. We have medical devices around here running Windows XP. How's that for a nightmare?
Seriously, ANY device can be infected with a new exploit whether it's up to date or not. New fully updated equipment is no less of a risk than old out of date equipment.
Keep it off the network. Or put away lots of money for the rainy day when it comes.
This is a lesson that should have been learned decades ago. That the question even needs to be asked just demonstrates how stupid the world has become.
My experience tells me that if my hardware is not running Debian, then at some point there will be no more updates.
And hackers is not the only problem, often the hardware just becomes useless.
E.g., I have a perfectly good old WiFI IP phone, but it only works on open networks or networks encrypted with WEP.
I have some devices that I would like to use to browse the internet. But they fail on websites with newer certificates.
Most dedicated systems like this does not belong on the internet, period. So unless there is some flaw or feature need, don't update and it will still work exactly as it did yesterday. And the day before, and the day before that.
Mechanical systems that keep, for example, trains from running into one another by tripping their brakes into full on, are well-understood. I took a course on doing the same thing in mixed hardware-software systems, so it's eminently possible.
The gotcha is you have to keep it really simple and run a validator like spin on it's protocol.
Most developers can do the spin part, but KISS? Distinctly less likely (;-))
davecb@spamcop.net
Implement a firewall with a small microcontroller with a relatively secure TCP/IP stack (ejip if you don't want to spend money, HCC embedded if you do) and do protocol level sanity checking and filtering of all network inputs.
How Do You Handle Hardware That Never Gets Software Updates?
Very carefully. (Buh-DUM-Tshhhh)
Borrowed from “How do porcupines make love?”
With apologies.
Our reign has gone on long enough. Indeed. Summon the meteors.
We have all sorts of insecure devices. There's no need to focus on IoT, or computers or electronics at all.
We have pickable locks, unbarred windows, windshield wipers, and high-speed cars separated by nothing but a strip of paint.
There's no reason to update devices that were never designed to change. We've gone centuries with devices that were never designed to change. You can steal a hammer. Does that mean hammer manufacturers need to implement security patches and thumb scanners to ensure that no one can hijack my hammer?
Start enforcing laws. Start arresting criminals.
A basic principle of security is least privilege. If a piece of outdated equipment needs to send udp packets on port 411 to a monitoring station, you set the firewall to allow it to send udp on port 411 to that particular station, and nothing else. If it doesn't need to take to web servers, you don't let it talk to web servers. You allow it to do only exactly what it needs to do.
Not sure what your equipment needs to do? You could check the manual, and otherwise open up Wireshark and set the filter to the IP of the equipment. Have a look at what it is sending and receiving. Then set the firewall to allow only exactly what is needed.
This is also an area where vlans come in very handy. Vlans act like completely separate networks, but they are configured within your switch, so a single 48-port switch can handle a dozen different, totally separate vlans.
Perhaps different parts of your network should be mostly separate, but you need to allow a little bit of specific communication between two vlans. That's when you plug a router or firewall into both vlans and set it to route only specifically allowed traffic between them. This doesn't even require two network ports - the same port can be in multiple vlans and the router can control traffic between vlans issuing a single cat6 cable. This is called "router on a stick".
If some of this went over your head, here's the simple version'
Call someone who has a CCNA Security certification or better (CCNP Security or CCIE Security). Tell them you're thinking about segregating different vlans and using an internal firewall to strictly control internal traffic. They'll get you set up.
Open source has taken off like wildfire. You are using it right now. You are just too dumb to realize it.
I use Slackware, along BSD, financially support projects that I use, and have followed the Linux community since Linus was still in college. It always amazes me how clueless the FOSS community is regarding issues such as this.
Just use Linux...
That's your fault for using M$..
etc.
For regulated systems, especially in pharma manufacturing, you are told what to use, how to use it, when to upgrade it, how to upgrade it, etc. Basically, once the system is certified by the FDA - you don't touch it - PERIOD. You purchase enough compute/control systems when you install it to last you through your production, which could be - 10, 15, 20+ years.
There is no, well, just upgrade to x - it's not allowed.
Before some equally clueless libertarian pinhead starts spouting off about 'over regulation' - stop and think for just one second what this system does. It controls the valves, temperatures, mixing, fermenting, refining, etc. of a chemical that people are to ingest. Where the difference between good and bad is measured in ppm, ppb, or even ppt depending on what's being made. Some endocrine chemicals are measured in 1/10ths or 1/100th of a ug!
Do you really want to apply patches to a system such as this? Doesn't matter that they are 'network', or 'mouse driver', or 'display' - the risk is WAY TOO GREAT to jack around with them.
Keep in mind that 'upgrades' require a new certification of that system, or depending on what it does, the entire production chain - which could run you a couple 10's of millions dollars.
So, before starting the typical FOSS rant, please have a clue of what you are talking about, first.
That doesn't help when a particular device from a particular manufacturer contains non-free software, as do the substitute devices from all competing manufacturers.
Devices running Windows XP are already unsupported. Devices run Windows 7 will be in the same boat as devices running Windows XP come January 2020.
I have multiple clients with non-networked computers. The oldest is running Windows 2000 (a Win98 system was retired a couple years ago). Security is not an issue if you don't network it. If you need to transfer files off it, use a USB flash drive or HDD which is used only for that purpose (i.e. you don't use it to copy music you've downloaded via filesharing).
If it must be networked, you can put it behind its own router. Rely on the router's firewall to protect it from outside intrusion (and of course don't do anything stupid like browse the web on it). I'm actually not very confident about this one because some random employee will undoubtedly try to use the system to login to their facebook account at some point. But the client absolutely insisted on networking some old XP computers so they could upload newly-recorded data files to Dropbox every night, and this was the best idea I could come up with.
and since i did a clean install of win7 on it more than a year ago it ever gets to connect to the internet, it does not even have the wifi password for the internet, but it does connect to the wifi a separate router that is LAN only, no internet on that router, it just runs some security cameras, so i can keep an eye on four different directions around the outside of my house, so if a hacker wanted to hack in to it they would have to be war driving right outside my house and nobody has done that
Politics is Treachery, Religion is Brainwashing
How often do you update your router? If your up time is over 60 days you are missing updates and are insecure.
I don't know any home/small business router company (TP-Link, Linksys, Netgear, ...) updating routers every 60 days. More like 1-2 times per year, for 1-2 years. And then nothing.
Perhaps you should look into Asus, which often updates at least quarterly, and often monthly:
* https://www.asus.com/Networking/RTAC68U/HelpDesk_BIOS/
* https://www.asus.com/microsite/2014/networks/routerfirmware_update/
And has been doing it for 4+ year-old products. Plus there is third-party code that leverages the GPL stuff that Asus releases:
* https://asuswrt.lostrealm.ca
* https://github.com/RMerl/asuswrt-merlin.ng
Agreed, alarming on a change in traffic makes sense, as does keeping a drive image of the system.
They buy it because it's better. It's better than Windows Phone (the first, second, theirs, and fourth attempts), it's better than Symbian, it's better than everything else people have tried. Why is it better? Linux is or reason it's better. Even Microsoft is using more and more Linux now. Is that because Microsoft has a religious zealotry for Linux? No, it's because Linux is better. Better than eating their own dog food.
>> Legacy software forcing people into Windows nowadays.
> Yeah, more than a billion people.
Yeah, legacy software has a LOT of people (companies, really) still stuck on Windows. Your point is?
The day I activated my current router, I put in a entry in the SysOp calendar saying "Router XYZ Active as of 20XX-XX-XX" with quarterly reminders.
I check the devices on those dates, or around those dates, and if it hasn't been updated in a year, I buy a replacement.
I do this for all the phones, tablets and other devices my family uses.
Yes, I use the word SysOp. I've been around that long.
This seems like excellent advice, and I see that a lot of the followups agree and provide some technical details. Still, I reckon a lot of owners of this old equipment may not have the technical know-how to do it right.
It seems to me somebody with appropriate energy and enterprise (which lets me out), could start a company providing just this kind of service.
In theory, theory and practice are the same; in practice they're different. (Yogi Berra & A. Einstein)
a device not connected to the network that just works is better than something doing untimely automatic updates.
Slashdot, fix the reply notifications... You won't get away with it...
realistically what are you going to do with high dollar customer made capital equipment that can't get a windows update? throw it out? no you keep using it until it breaks.
Scott
Windows 7 gets free security updates until some time in 2020, according to the linked article. The 2015 date is for desktop support. Plus the Windows 7 embedded manufacturers get 10 years of support after the end-of-lifetime for the OS (not sure when that was).
I was personally very upset when Motorola refused to provide me a software update for a device, designed for both long-term and short-term use!
It was an SN74LS139N Motorola Dual Decoder 2-4 Line Plastic TTL chip.
How dare they deny me software updates for this chip containing two inverters and four AND gates!
I don't give a damn that they designed it for embedded use, I should be able to update the software running on it!
Right?
1. buy only well supported or open devices
2. (if you can't do that,) do not connect them to a network
3. if you must connect them to a network, make it a private network, make sure it is properly setup, closing all ports by default
4. if you can't have them on a private network and they must connect to your lan or worse, internet - hope for the best.
On a long enough timeline, the survival rate for everyone drops to zero.
Outside of Apple, Google phones and maybe one or two, you are LUCKY to get any updates.
I have a perfectly good HP Scanner I bought years ago. Still works fine, but only on XP with the software and on Windows 7 using the Windows tools; HPs software doesn't work on Windows 7. I have a Virtual Machine running Windows XP just so I can keep using my perfectly good HP Scanner and my perfectly good Sony HandyCam which also only works on XP.
[John]
Shit better not happen!
Comment removed based on user account deletion
I put the cans and bottles in, and take the receipt to the cashier. (Bottle return machines in this area still run Windows 98. Yes, I did say 98.) Except recently, the machines have been so unreliable that I've just been throwing the containers away and taking a hit on the deposit. I don't see it getting any better, because there's very little financial reason for stores to take bottles back.
I'm told by someone who services them, that a lot of POS machines are still running Windows 98. Just exactly the place you want an old, unpatched OS.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
"I'm sorry. We no longer support that equipment. I'll be happy to connect you with sales to purchase a new model."
Uh, yeah. It's a quarter million dollar piece of lab equipment that's 6 years old and you want us to just buy a new one in a time of tight grants.
I throw away the device and make a note to never again buy one of their products.
Allowing updates, that have not been carefully checked, breaks the equipment. The equipment can not be allowed to break, or people may die.
Do not connect the equipment to any cable that goes outside. In some installations that even includes power. In many of the installations in question, they have always been that way, anyway.
If you have to get on the internet, use your cellphone, not the equipment. Connecting critical stuff to the entire world is crazy...
Manufacturers are screaming about updates, because they can make money from them. Most of the (Windows) updates have nothing to do with security, though. Also, the new CPU faults mean that you are not secure even with updates, until you can get a new (fixed) CPU chip.
Buying new equipment is not a choice, because it is not available for any price. Or sometimes it is so different, that it can not be used within a reasonable time.
If you are just talking about your home router, that's different. Trash it and get a new one that you can configure properly.
My 2017 Honda Civic is running Debian. It is so outdated that even the app the comes with it won't update or run. I went to the dealership and I might as well have gone to the grocery store. All I got was blank stares. I wrote Honda and got nothing back. I fear many of these cars that have smart consoles are just IoT devices waiting to be exploited.
Yes, I tried updating it but Honda, like Samsung to phones, has their own flavor.
"If you are on fire you can just stop, drop, and roll. If you fall into Lava you are just dead." - my 5yr old daughter
If it is your router, you should look at OpenWRT, DD-WRT or similar alternative Linux router distributions. This way you can ensure the updates for many years. (Or better, you can invest in a low-power PC and just install pf-sense, which would be leaps and bounds more capable than a tiny ARM machine).
If it is a security system, camera, alarm, etc. you'd need to make sure they are in a separate network. If possible a distinct network for each and every device with proper router rules, so that for example your NVR recording hub can access the camera only, etc. Of course Internet access should be disabled, since they usually happen to connect to a backdoor server. You might have a temporary rule to update firmware if necessary, or setup a VPN to access for your mobile devices. (Both iOS and Android can be setup to connect to your pf-sense router from outside of your LAN, Windows and Mac require some more steps due to self signed certificate issues).
If it is a "smart" device like a fridge, thermostat, or a light controller which needs to connect to the internet, you can keep them in a separate network to minimize damage, and replace them, as soon as the manufacturer stops updating. Unfortunately it might not be easy to letting to of a functional smart power outlet with monitoring, etc. but if there are known un-patched holes, you would not want the entire internet to be able to cause damage to whatever equipment you have down that connector.
Overall use your own judgment, and learn networking basics. Also do not be lazy (I still have some cameras that I need to finish securing).
I would be very surprised if you could demonstrate the existence of any modifiable software running on the internal processors of the inverters and the AND gates that has not been made freely available under extremely permissive licence.
I'd be surprised if you could demonstrate the same thing for embedded devices not designed to be modified or updated by users -- no matter their complexity.