Slashdot Mirror
Ask Slashdot: How Do You Handle Hardware That Never Gets Software Updates? (hpe.com)
New submitter pgralla writes from a report via HPE: Many devices, designed for both long-term and short-term use, were shortsighted when it came to flexibility. How do you handle the hardware that never gets software updates, such as embedded systems and task-dedicated equipment? The article that pgralla shared provides the example of medical devices running Windows 7. "Many of the current generation, when they were first released, used Windows 7, and the devices still work well enough that they remain in service today," reports HPE. "But Microsoft ended mainstream support for Windows 7 back in January 2015, so the operating system gets updated only with an occasional security patch as part of Microsoft's extended support. In January 2020, that extended support will end as well." Many IoT devices are in a similar boat as they're powered by embedded Linux and are not designed to be updated after they enter service."
Of course, these outdated devices create all sorts of security concerns. "Hackers and their access to knowledge and computing power only go up as the years pass, which means that long-lived, fixed-firmware devices become ever more insecure over time," says Michael Barr, founder of the Barr Group, which provides engineering and consulting services for the embedded systems industry. The WannaCry ransomware hack in 2017 affected not just PCs but also medical devices, and ended up costing businesses $4 billion.
Of course, these outdated devices create all sorts of security concerns. "Hackers and their access to knowledge and computing power only go up as the years pass, which means that long-lived, fixed-firmware devices become ever more insecure over time," says Michael Barr, founder of the Barr Group, which provides engineering and consulting services for the embedded systems industry. The WannaCry ransomware hack in 2017 affected not just PCs but also medical devices, and ended up costing businesses $4 billion.
With a single shot of whiskey.
-Clutch Nixon
....don't buy it.
I've seen SO many people whining about MS' forced reboots, etc. STOP!
If there is not a sensible option available, demand that your vendor make a version that can be sensibly updated. Too many purchasing decisions just don't have any sensible criteria. ("Oh, it's built on Win XP and you aren't updating it? OK - scratch!")
Many old tools are computer based
Some old CNC machines run on MS-DOS and a 286 processor
As long as the hardware stays alive, they continue to do the job
If they must be networked, restrict their access to the local net
I have a number of Rohde and Schwarz FSEB and FSEA spectrum analyzers. These cost at least $80,000 new (I bought them used for a few thousand at most). They come with an old version of windows. I similarly have other electronic test equipment with old Windows or even old Linux which the manufacturer doesn't update any longer. For the Linux-based ones I could hack in a new Linux and make it use the old ABI, forget about Windows.
But what really clued me in was that the Rohde and Schwarz equipment had a battery soldered on the CPU board, and it was an hour-and-a-half service to get to it. A lot of stuff had to be removed.
Similarly, my Tektronix 500-series oscilloscopes had two 40-pin DIP Dallas Semiconductor battery-backed memory and clock chips. The batteries in these die and they aren't socketed. When the batteries die, the 'scopes lose their calibration. The company won't give you the program to recalibrate them.
The manufacturers just want you to buy new ones.
So, obviously I back SDR-based test equipment that's Open Source. Who needs a company that wants to screw you?
Bruce Perens.
Medical devices with Windows 7? That's a laugh. We have medical devices around here running Windows XP. How's that for a nightmare?
Most of that consumer crap is just that. Do not buy it. If you want or need to be connected get a nice laptop and/or smartphone from a reputable vendor that provides regular updates and donâ(TM)t buy the crap. Do not be suckered by shiny.
Since it is open source, just update it yourself or pay someone to update it for you. That is why you have the source code.
Seriously, ANY device can be infected with a new exploit whether it's up to date or not. New fully updated equipment is no less of a risk than old out of date equipment.
Keep it off the network. Or put away lots of money for the rainy day when it comes.
This is a lesson that should have been learned decades ago. That the question even needs to be asked just demonstrates how stupid the world has become.
My experience tells me that if my hardware is not running Debian, then at some point there will be no more updates.
And hackers is not the only problem, often the hardware just becomes useless.
E.g., I have a perfectly good old WiFI IP phone, but it only works on open networks or networks encrypted with WEP.
I have some devices that I would like to use to browse the internet. But they fail on websites with newer certificates.
Only permit approved tcp/udp communication within your infrastructure. It's not optimal but a good step. Ban automatic settings of a default gateway in your DHCP reservations for these devices and set a useful or unusable default gateway. Better yet use MAC filtering on DHCP reservations to capture these devices and keep them jailed properly.
Then, after all that.. do a trade study to see what vendors provide long-term support for devices you need for your organizations.
It's all a crap-shoot you are forced to find an uncomfortable bottom line.
Peace out.
Most dedicated systems like this does not belong on the internet, period. So unless there is some flaw or feature need, don't update and it will still work exactly as it did yesterday. And the day before, and the day before that.
Mechanical systems that keep, for example, trains from running into one another by tripping their brakes into full on, are well-understood. I took a course on doing the same thing in mixed hardware-software systems, so it's eminently possible.
The gotcha is you have to keep it really simple and run a validator like spin on it's protocol.
Most developers can do the spin part, but KISS? Distinctly less likely (;-))
davecb@spamcop.net
Yes, sir!!!
- creimer
Implement a firewall with a small microcontroller with a relatively secure TCP/IP stack (ejip if you don't want to spend money, HCC embedded if you do) and do protocol level sanity checking and filtering of all network inputs.
It means it's well designed in the first place. I still keep my NES, SNES, Atari, Mega Drive, PlayStation, Saturn, Gameboy Advance, etc etc around and use them and I'm happy that every time I turn them on or start a game that I don't have to wait for some update to finish downloading and installing.
How Do You Handle Hardware That Never Gets Software Updates?
Very carefully. (Buh-DUM-Tshhhh)
Borrowed from “How do porcupines make love?”
With apologies.
Our reign has gone on long enough. Indeed. Summon the meteors.
We have all sorts of insecure devices. There's no need to focus on IoT, or computers or electronics at all.
We have pickable locks, unbarred windows, windshield wipers, and high-speed cars separated by nothing but a strip of paint.
There's no reason to update devices that were never designed to change. We've gone centuries with devices that were never designed to change. You can steal a hammer. Does that mean hammer manufacturers need to implement security patches and thumb scanners to ensure that no one can hijack my hammer?
Start enforcing laws. Start arresting criminals.
HTC made one update to an early Android tablet and several unfulfilled promises for further updates. I wrote them off forever and stopped using their tablet when it was clear they weren't going to follow though.
How the Fuck can you expose your self as the TOOOL you are??
republishing corporate papers, wiki's, and community crap.
you fucking dork..
I guess msmash puled hers out and clouded your mental craptastic abilities..
like JollyRancher,
Keep On Suckin'
ya twisted, FuckTard..
Do you suck Everyone;s dick with that mouth? Or just msmash's?
A basic principle of security is least privilege. If a piece of outdated equipment needs to send udp packets on port 411 to a monitoring station, you set the firewall to allow it to send udp on port 411 to that particular station, and nothing else. If it doesn't need to take to web servers, you don't let it talk to web servers. You allow it to do only exactly what it needs to do.
Not sure what your equipment needs to do? You could check the manual, and otherwise open up Wireshark and set the filter to the IP of the equipment. Have a look at what it is sending and receiving. Then set the firewall to allow only exactly what is needed.
This is also an area where vlans come in very handy. Vlans act like completely separate networks, but they are configured within your switch, so a single 48-port switch can handle a dozen different, totally separate vlans.
Perhaps different parts of your network should be mostly separate, but you need to allow a little bit of specific communication between two vlans. That's when you plug a router or firewall into both vlans and set it to route only specifically allowed traffic between them. This doesn't even require two network ports - the same port can be in multiple vlans and the router can control traffic between vlans issuing a single cat6 cable. This is called "router on a stick".
If some of this went over your head, here's the simple version'
Call someone who has a CCNA Security certification or better (CCNP Security or CCIE Security). Tell them you're thinking about segregating different vlans and using an internal firewall to strictly control internal traffic. They'll get you set up.
I think they are referring to XP not Windows 7?
9x9s uptime! Avoid creating broken software with the normal programming languages that cause years of software maintenance woes!
If a device runs embedded software, isn't connected to an open network, and does what it's supposed to do, I just keep using it.
I use Slackware, along BSD, financially support projects that I use, and have followed the Linux community since Linus was still in college. It always amazes me how clueless the FOSS community is regarding issues such as this.
Just use Linux...
That's your fault for using M$..
etc.
For regulated systems, especially in pharma manufacturing, you are told what to use, how to use it, when to upgrade it, how to upgrade it, etc. Basically, once the system is certified by the FDA - you don't touch it - PERIOD. You purchase enough compute/control systems when you install it to last you through your production, which could be - 10, 15, 20+ years.
There is no, well, just upgrade to x - it's not allowed.
Before some equally clueless libertarian pinhead starts spouting off about 'over regulation' - stop and think for just one second what this system does. It controls the valves, temperatures, mixing, fermenting, refining, etc. of a chemical that people are to ingest. Where the difference between good and bad is measured in ppm, ppb, or even ppt depending on what's being made. Some endocrine chemicals are measured in 1/10ths or 1/100th of a ug!
Do you really want to apply patches to a system such as this? Doesn't matter that they are 'network', or 'mouse driver', or 'display' - the risk is WAY TOO GREAT to jack around with them.
Keep in mind that 'upgrades' require a new certification of that system, or depending on what it does, the entire production chain - which could run you a couple 10's of millions dollars.
So, before starting the typical FOSS rant, please have a clue of what you are talking about, first.
I was called in the get an old dilatometer to work (measures coefficient of thermal expansion). It ran on Windows 3.1. It was just a bad battery, and the BIOS forget the hard drive. Only charged an hour plus travel time. They were pleased as punch that it was still working, but they didn't accept my proposal to port things to a more modern system..
I have multiple clients with non-networked computers. The oldest is running Windows 2000 (a Win98 system was retired a couple years ago). Security is not an issue if you don't network it. If you need to transfer files off it, use a USB flash drive or HDD which is used only for that purpose (i.e. you don't use it to copy music you've downloaded via filesharing).
If it must be networked, you can put it behind its own router. Rely on the router's firewall to protect it from outside intrusion (and of course don't do anything stupid like browse the web on it). I'm actually not very confident about this one because some random employee will undoubtedly try to use the system to login to their facebook account at some point. But the client absolutely insisted on networking some old XP computers so they could upload newly-recorded data files to Dropbox every night, and this was the best idea I could come up with.
and since i did a clean install of win7 on it more than a year ago it ever gets to connect to the internet, it does not even have the wifi password for the internet, but it does connect to the wifi a separate router that is LAN only, no internet on that router, it just runs some security cameras, so i can keep an eye on four different directions around the outside of my house, so if a hacker wanted to hack in to it they would have to be war driving right outside my house and nobody has done that
Politics is Treachery, Religion is Brainwashing
How often do you update your router? If your up time is over 60 days you are missing updates and are insecure.
I don't know any home/small business router company (TP-Link, Linksys, Netgear, ...) updating routers every 60 days. More like 1-2 times per year, for 1-2 years. And then nothing.
Perhaps you should look into Asus, which often updates at least quarterly, and often monthly:
* https://www.asus.com/Networking/RTAC68U/HelpDesk_BIOS/
* https://www.asus.com/microsite/2014/networks/routerfirmware_update/
And has been doing it for 4+ year-old products. Plus there is third-party code that leverages the GPL stuff that Asus releases:
* https://asuswrt.lostrealm.ca
* https://github.com/RMerl/asuswrt-merlin.ng
Agreed, alarming on a change in traffic makes sense, as does keeping a drive image of the system.
They buy it because it's better. It's better than Windows Phone (the first, second, theirs, and fourth attempts), it's better than Symbian, it's better than everything else people have tried. Why is it better? Linux is or reason it's better. Even Microsoft is using more and more Linux now. Is that because Microsoft has a religious zealotry for Linux? No, it's because Linux is better. Better than eating their own dog food.
>> Legacy software forcing people into Windows nowadays.
> Yeah, more than a billion people.
Yeah, legacy software has a LOT of people (companies, really) still stuck on Windows. Your point is?
The day I activated my current router, I put in a entry in the SysOp calendar saying "Router XYZ Active as of 20XX-XX-XX" with quarterly reminders.
I check the devices on those dates, or around those dates, and if it hasn't been updated in a year, I buy a replacement.
I do this for all the phones, tablets and other devices my family uses.
Yes, I use the word SysOp. I've been around that long.
Wow, most posts so far are about Windows and its crappy updates. Hardware goes much wider than laptops. That Garmin/Navman GPS - good luck updating an older model - they want you to buy a new one. And cars now have navigation - needs updates. Example - Honda CRV: Honda NZ want $400 to update the maps. The car will now have old maps forever. Imported cars from Japan have nav systems and the local dealerships REFUSE to update or switch them to NZ. Even though you see on Youtube that it is just a DVD and globally available (add secret sauce to install it though). For some $$ you can find things from dodgy suppliers, but your maps might be 4-6 years behind. Next - smartphones. Once uncle google decides your Android is too old, bye bye. And so many phones are locked to a network - both the network and the phone maker (eg Samsung) do NOT supplu updates. Cameras get firmware updates, but only for so long, and 99% of the time they assume (assume makes a fool out of you and me) you have Windows installed. And then you get crippleware products like Canon scanners - drivers are for Windows only, so you cannot do the advanced stuff like scan film/negs from Linux. In the end, we are buying future landfill as manufacturers want to sell more new stuff, not support their old gadgets.
I haven't met a linux device that can't be updated. On the other hand there are countless Windows CE devices (remember Windows CE?) still being sold as new - especially GPS Navigators - that will never ever see any updates whatsoever.
This seems like excellent advice, and I see that a lot of the followups agree and provide some technical details. Still, I reckon a lot of owners of this old equipment may not have the technical know-how to do it right.
It seems to me somebody with appropriate energy and enterprise (which lets me out), could start a company providing just this kind of service.
In theory, theory and practice are the same; in practice they're different. (Yogi Berra & A. Einstein)
a device not connected to the network that just works is better than something doing untimely automatic updates.
Slashdot, fix the reply notifications... You won't get away with it...
realistically what are you going to do with high dollar customer made capital equipment that can't get a windows update? throw it out? no you keep using it until it breaks.
Scott
Windows 7 gets free security updates until some time in 2020, according to the linked article. The 2015 date is for desktop support. Plus the Windows 7 embedded manufacturers get 10 years of support after the end-of-lifetime for the OS (not sure when that was).
I was personally very upset when Motorola refused to provide me a software update for a device, designed for both long-term and short-term use!
It was an SN74LS139N Motorola Dual Decoder 2-4 Line Plastic TTL chip.
How dare they deny me software updates for this chip containing two inverters and four AND gates!
I don't give a damn that they designed it for embedded use, I should be able to update the software running on it!
Right?
Safe to assume all software comes out new with weaknesses and requires fixes. Probably the best way to deal with it, is do not connect directly to the internet or limit exposure and don't buy any equipment that won't receive good support when a flaw is found. Even the most simple software can be a exposed threat. Equipment not updated should never be directly exposed to networks.
1. buy only well supported or open devices
2. (if you can't do that,) do not connect them to a network
3. if you must connect them to a network, make it a private network, make sure it is properly setup, closing all ports by default
4. if you can't have them on a private network and they must connect to your lan or worse, internet - hope for the best.
On a long enough timeline, the survival rate for everyone drops to zero.
In many cases you will find that the device only need to receive data from the network, or only needs to send data to something else. A data diode is then a very good option.
An old windows box will have hugely complex services SMB, RPC etc open, it will have a full blown web browser installed etc.
An old embedded linux box shouldn't have any of this, it will have whatever service the embedded device requires, and possibly a management service like http or ssh...
If properly built in the first place, a linux system (or other embedded os) should require far less ongoing updates due to a much smaller attack surface. Not invulnerable by any means, but hugely preferable to the windows approach.
Most of the compromised Linux IOT devices out there are actually compromised via default passwords, which can happen to any system no matter how well updated it is.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
This is by design so that every few years you have to buy new hardware. And this goes way back to when a Phoebus cartel decided that the light bulbs should have a limited life span instead of being able to be used for years at a time.
Think 80's era Peter Gabriel song/music video.
SLEDGEHAMMER!!!!!!!!!!!!!
Outside of Apple, Google phones and maybe one or two, you are LUCKY to get any updates.
I have a perfectly good HP Scanner I bought years ago. Still works fine, but only on XP with the software and on Windows 7 using the Windows tools; HPs software doesn't work on Windows 7. I have a Virtual Machine running Windows XP just so I can keep using my perfectly good HP Scanner and my perfectly good Sony HandyCam which also only works on XP.
[John]
Shit better not happen!
Comment removed based on user account deletion
Seriously, the correct answer is: you don't buy it in the first place. Obviously, some people (even me) make occasional exceptions, but whenever that happens, the hardware in question is assumed to be totally disposable.
This isn't so much a policy all its own, as it's part of the more general strategy of making sure that you have guaranteed long-term maintenance for anything and everything that you personally consider "important."
Most of the time, the way you do things is that you buy generic hardware and then install Free Software on it, so that you know what it has, and you're in control. "You're in control" are the keywords in every single computer-related decision. If you're not in control, then it's disposable.
I have three somewhat-expensive disposable pieces of hardware (where I am not in control of the software and am totally at a vendor's mercy for whatever future updates I ever get, if any), and they're all about the same price:
1) Samsung Galaxy J7 phone (about $225)
2) some mid-range nvidia graphics card (don't remember the model #, but it's from 2016) that was about $200. Runs proprietary drivers. There is no telling how long those drivers will keep working.
3) buffalo AP/router (was maybe $150-$200 in 2010? don't remember)
So I guess around $250 is the most I ever spend on any hardware, where I have no control over the availability of software maintenance and updates; that's apparently the largest wad of cash that I would be willing to touch a burning match to. (Though my total risk is around $600-$700 for the entire house.)
Your personal threshold for how much power you are willing to let total strangers have over you, will be different. Maybe $50 is the most you'll pay. Apple and Microsoft customers will risk thousands of dollars! It's all up to you. But the point is, this is something you worry about before you spend anything, not something you address later. It's a critical aspect of the initial purchase. I would even go so far as to say it dominates the initial purchase, or is at least in the same league as the price tag.
This is very sad and would only happen to someone who simply doesn't care:
These people are doing it wrong. Your embedded Linux devices should be pi or beagleboards or something, running software that you loaded. If you loaded it once, you can load an update again, later. OTOH if you're not in control then you're not in control. So throw it away, since it was disposable.
I put the cans and bottles in, and take the receipt to the cashier. (Bottle return machines in this area still run Windows 98. Yes, I did say 98.) Except recently, the machines have been so unreliable that I've just been throwing the containers away and taking a hit on the deposit. I don't see it getting any better, because there's very little financial reason for stores to take bottles back.
I'm told by someone who services them, that a lot of POS machines are still running Windows 98. Just exactly the place you want an old, unpatched OS.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
I don't know if you've seen Windows in the last 10-15 years, but holy crap. My wife brought home a Windows laptop from work, for whenever she suddenly has to do remote work at night or on a weekend (some kind of proprietary VPN and VNC like thing). This happens every few months.
Guess what it's like to use a Windows computer every few months. (Seriously, if you don't know what it's like, then you would never guess.) After you turn on the computer, it can sometimes literally take hours before you're allowed to use it. I am not making this up. Microsoft punishes people who apply updates; it's way harder on the user to update, than it is for users of normal OSes. My point being, you're right that Window users don't update, but that's because they're trained to not update, and any deviation from the training is dealt with harshly, with the apparently intent of inflicting as much inconvenience and discouragement as possible upon the user.
If those same people had a normal OS, I bet they'd update much more often.
Windows is a special case. Never forget that. When you talk about the masses, even though that's a lot of people, it's still not normal. Most computers aren't like that. Any modern computer and OS isn't anything like that. I think an average mouthbreather can keep things updated; they just don't because they happen to be stuck with particularly bad legacy system. It's about that particular platform, not the users.
i have many devices the manufactures have abandon,i'm tired of buying stuff that is still functional except some greedy irresponsible corporation decided to walk away from the commitments and customers. There's no such thing as corporate integrity, these irresponsible people make out like bandits and then take the money and run. People are getting cheated and screwed and yet nothing ever changes. The rich rip people off and get away scot free. The whole system is completely corrupt and ineffective. If people weren't so complacent and uncaring, they'd never put up with this evil crap.
Please, stop letting yourself, and everyone else, from get constantly fucked over by corporations. Corporate reform to remove lack of culpability is desperately required.
"I'm sorry. We no longer support that equipment. I'll be happy to connect you with sales to purchase a new model."
Uh, yeah. It's a quarter million dollar piece of lab equipment that's 6 years old and you want us to just buy a new one in a time of tight grants.
I work in a company that uses manufacturing equipment with various operating systems, as old as Windows NT, which don't get updates. It's great. We've never had a production outage, from a Windows NT update hosing the system or breaking compatibly with the attached hardware.
The desktop computers, on the other hand, have been hell to work with. We have started migrating everything to Linux, because the updates don't break things, but for the computers running Windows, we occasionally have to stop production until a computer gets rolled back, because an update broke something.
I throw away the device and make a note to never again buy one of their products.
No updates for your working iThingie?
Then it still works the same as when you bought it!
Adults know you do not need updates unless you are having a problem which is corrected by an update.
In Windows there have been no updates which FIXED something for a Person using the computer since WinXP.
That you cannot read about, and mitigate the horrible security of Windows without updates is pathetic.
Air gap your special function equipment and keep using it. Cheap and it works. Back it up. If you really must exchange files on media, do so with a modern, updated, and protected machine
I have another question:
How do you handle operating systems that update itself and then ditch your perfectly functioning hardware, such as windows 10?
Allowing updates, that have not been carefully checked, breaks the equipment. The equipment can not be allowed to break, or people may die.
Do not connect the equipment to any cable that goes outside. In some installations that even includes power. In many of the installations in question, they have always been that way, anyway.
If you have to get on the internet, use your cellphone, not the equipment. Connecting critical stuff to the entire world is crazy...
Manufacturers are screaming about updates, because they can make money from them. Most of the (Windows) updates have nothing to do with security, though. Also, the new CPU faults mean that you are not secure even with updates, until you can get a new (fixed) CPU chip.
Buying new equipment is not a choice, because it is not available for any price. Or sometimes it is so different, that it can not be used within a reasonable time.
If you are just talking about your home router, that's different. Trash it and get a new one that you can configure properly.
My 2017 Honda Civic is running Debian. It is so outdated that even the app the comes with it won't update or run. I went to the dealership and I might as well have gone to the grocery store. All I got was blank stares. I wrote Honda and got nothing back. I fear many of these cars that have smart consoles are just IoT devices waiting to be exploited.
Yes, I tried updating it but Honda, like Samsung to phones, has their own flavor.
"If you are on fire you can just stop, drop, and roll. If you fall into Lava you are just dead." - my 5yr old daughter
If it is your router, you should look at OpenWRT, DD-WRT or similar alternative Linux router distributions. This way you can ensure the updates for many years. (Or better, you can invest in a low-power PC and just install pf-sense, which would be leaps and bounds more capable than a tiny ARM machine).
If it is a security system, camera, alarm, etc. you'd need to make sure they are in a separate network. If possible a distinct network for each and every device with proper router rules, so that for example your NVR recording hub can access the camera only, etc. Of course Internet access should be disabled, since they usually happen to connect to a backdoor server. You might have a temporary rule to update firmware if necessary, or setup a VPN to access for your mobile devices. (Both iOS and Android can be setup to connect to your pf-sense router from outside of your LAN, Windows and Mac require some more steps due to self signed certificate issues).
If it is a "smart" device like a fridge, thermostat, or a light controller which needs to connect to the internet, you can keep them in a separate network to minimize damage, and replace them, as soon as the manufacturer stops updating. Unfortunately it might not be easy to letting to of a functional smart power outlet with monitoring, etc. but if there are known un-patched holes, you would not want the entire internet to be able to cause damage to whatever equipment you have down that connector.
Overall use your own judgment, and learn networking basics. Also do not be lazy (I still have some cameras that I need to finish securing).
I would be very surprised if you could demonstrate the existence of any modifiable software running on the internal processors of the inverters and the AND gates that has not been made freely available under extremely permissive licence.
I would be very surprised if you could demonstrate the existence of any modifiable software running on the internal processors of the inverters and the AND gates that has not been made freely available under extremely permissive licence.
I'd be surprised if you could demonstrate the same thing for embedded devices not designed to be modified or updated by users -- no matter their complexity.