Slashdot Mirror
The Coming IT Nightmare of Unpatchable Systems
snydeq (1272828) writes "Insecure by design and trusted by default, embedded systems present security concerns that could prove crippling if not addressed by fabricators, vendors, and customers alike, InfoWorld reports. Routers, smart refrigerators, in-pavement traffic-monitoring systems, or crop-monitoring drones — 'the trend toward systems and devices that, once deployed, stubbornly "keep on ticking" regardless of the wishes of those who deploy them is fast becoming an IT security nightmare made real, affecting everything from mom-and-pop shops to power stations. This unpatchable hell is a problem with many fathers, from recalcitrant vendors to customers wary of — or hostile to — change. But with the number and diversity of connected endpoints expected to skyrocket in the next decade, radical measures are fast becoming necessary to ensure that today's "smart" devices and embedded systems don't haunt us for years down the line.'"
Wait until we have driverless cars on the road. But I'm sure they'll all be bullet-proof secure, don'tcha think?
The lesson wasn't learned, but the problem was somewhat mitigated. Big software companies adopted regular patch cycles and deployed patch management tools on their customers. It kinda worked because PC are powerful computers well designed to be upgraded and modified.
This is not the case for many embedded systems. They are designed to be installed and then you forget about them. So the "classic" mitigation technique doesn't work. This is a big problem.
Well, that would be less of a problem if you didn't surf SlashDot using your refrigerator or crop-monitoring drone...
There are two lessons here: one, if you make something non-upgradeable it may have a bug that requires a fix; two, if you make something upgradeable some nefarious actor could exploit that and install something bad.
"Poorly designed", or "incorrectly designed" - perhaps. I'm fairly sure that even the ATM designers who went with an embedded MicroSoft operating system felt that they had mediated security risks adequately to deploy their systems. Incidentally, I had a chance to peek inside a local casino's slot machines - all of them, regardless of external appearance were based on an identical piece of hardware. Watching them boot showed me a MicroSoft OS underlying those slots. Not a problem, as I'm fairly certain that none of the slot machines on the floor have any conceivable way of ever connecting directly to any network except for the dark wire casinos use for exactly this purpose.
My takeaway point is that the summary is (IMHO) slightly biased. The original article appears to be well written. Just to ask - how many embedded systems should be permitted to ever connect to the internet? ATM's, for example, should demonstrably be either confined to a darknet or (as I've seen in some places) required to use dialup access. It's not perfect, but it adds a significant obstacle for crackers to overcome. The casino I mentioned earlier seems to get this point.
I don't mind smart appliances - but again, I don't see why they need internet access. The exceptions to this (smart TV's, for example) should be viewed with suspicion specifically because they are likely to be connected to the internet in some way, but my smart refrigerator probably shouldn't be - and ATM's, slot machines, SCADA systems, etc. almost certainly should never be.
Different nightmare. The Y2K embedded system nightmare was systems that wouldn't know what to do when the clock rolled over. By and large, the doomsayers were completely wrong. The current problem is *Internet enabled* embedded systems, easily hackable, out of warranty, out of support, manufacturer TU, owner/deployer isn't even sure how many they have, or where they're located, etc., etc. Picture making a botnet out of all the traffic light controllers, or the elevator controllers, or smart water meters, or internet toasters.
You know, I'll bet if you fixed your hosts file...
There are two bleeding edges. One is the leading edge of cutting technology.
There other is the trailing edge where systems age out because they take a lot of effort to update.
One way the trailing edge can not be updated because the overall system is designed to where there are critical parts that can not be monkeyed with in a low risk scenario. (This does happen).
The other option on the trailing edge is where the systems are not worth the effort. Most of the Internet of Everything appliances really have zero income after the first few months and yet are expected to have a longer lifetime than many major IT infrastructure requirements.
They are designed to be installed and then you forget about them. So the "classic" mitigation technique doesn't work. This is a big problem.
Hell, I thought the "classic" mitigation schemata for embedded devices was to not have them networked at all, leaving them to run for years (decades?) on end.
(See also the hordes of NT Telecom PBXes out there which are likely still around, requiring a goofball proprietary connection to a computer running OS/2 (!?) in order to patch it (or more commonly, you did it to add new/licensed features or to fix something gone corrupt).)
Therein lies the whole problem with the paradigm, truth be told - originally, embedded devices didn't communicate with jack shit - you unpacked it, turned it on, maybe configured it, and then you forget that it existed until it broke (at which time the vendor/contractor sent someone out to fix it), or got replaced.
All that said, hell, we already have a testbed for this nightmare - an ocean of smartphones whose carriers and manufacturers ceased to give a crap whether their wares ever got upgraded.
Quo usque tandem abutere, Nimbus, patientia nostra?
Probably not unless the user wants it fixed, and most don't. People have plenty of experiences with patches breaking new things, or taking away old functionality they had come to depend on. When someone tells me "this patch will solve all your problems", they usually aren't advertising the list of new problems they're creating for me. Anyone who plays iPhone app games knows that the patches sometimes come with game-stopping bugs; other patches have been known to suddenly add annoying advertising.
Usually, I'm at a point of equilibrium where I am at least coping with the bugs in the devices surrounding me. If I know that the "mute button" on my GoogleTV box doesn't work unless I press it twice, I simply learn to press it twice; while I know it's a stupid workaround, it's one I can live with. What I might not be able to live with are the bugs that come with the next round of patches.
Now, we make that experience hurdle even harder to scale: as a end user, I think security patches are worse than regular patches. The end user doesn't see a physical benefit from the patches, but knows he might suffer. What does he care if his thermostat or washing machine is sending spam around the world, as long as his house is warm and his clothes are clean? But if he installs the patches, he risks having a cold house or dirty clothes, or even advertisements streaming across his refrigerator's screen. It's just not worth the risk to patch them.
And if you want to see a really risk-averse, don't-patch-me crowd, talk to the SCADA industrial control people. If you suggest you need to update the software running the sewage ejection pump, the city engineer is going to hand you an invoice for $20,000 and say "that covers my cost of testing your patch."
John
Well as an Embedded System Designer I have to speak up here, systems are usually not insecure because of lazy development, systems are insecure because clients, managers and stakeholders don't provide proper funding, deadlines or requirements. The number of times I've had to go to a manager or project manager and ask them to clarify a customers request is almost sad. The amount of times I've had to go to the same group and ask for twice or three times the amount of time to develop a solution is almost sad and the amount of times I've had to ask for much more funding to do a proper job is sad. For some unknown reason embedded designers aren't treated like normal software developers and the truth is we aren't. We don't rely on some insecure patched to hell OS to keep us safe and we don't trust laughable memory managers and kernels to keep us crash free and running smooth. We do the real work in the development world and generally it's the GUI designer who takes the credit.
We generally don't work in the world of garbage collected and managed languages, we don't work in the world where everything is already setup and ready to be called through some piss poor abstracted class implementation of system.IO and we don't get safety nets under us to catch what falls through in some kind of completely illogical and messed up exception error system ( C# ). To say embedded systems are insecure is really another way to say one of several things:
1. You didn't allocate enough time, money or proper requirements.
2. You didn't hire someone who is qualified to the job, such as putting a desktop developer onto an embedded project.
3. You didn't consider security when you dreamed up you're fragmented and broken project idea.
This is of course mitigated by a great developer who will go back to the table of executives and tell them they need what they need and won't start until it's delivered. You can't treat an embedded project like a normal software project, when you do you'll end up with systems that make Microsoft proud ( aka 0 security and patch opportunities to fly to the moon ), you need to treat an embedded project like an embedded project and give the embedded developer what he / she needs. Doing other wise will always end up you shit creek and generally the manager or stakeholder is left with the paddle looking like a fool.
<RANT>
One thing that's causing problems is the habit of Apple and Microsoft to abandon operating systems for new, often incompatible ones, instead of fixing the bugs in them. OSX 10.6.8 is full of problems; the only way to fix them is to move up to OSX 10.7 or further, which in turn can break a lot of things, because the later release isn't just fixed (if, in fact, it is fixed), it's a different animal altogether. Just one example. OS vendors take the view that you can either move forward with them, or die in a fire. Windows, Ubuntu, XP, etc... same deal.
I'm not saying these old OS's should get new features. But bugs? They should be fixed as long as humanly possible. The product was sold as having feature set X, and working. If it doesn't work as advertised, or is unreliable, it shouldn't be abandoned, it should be fixed. Except in the very rare case where it is not possible (I can't even think of one of those, actually.)
The problem is multifaceted. It isn't just that users are left with a choice of being left behind and becoming steadily more vulnerable to exploits; it is also that as the OS vendors keep jumping away from their buggy versions, the OS landscape, as it were, is left lettered with broken junk, and the new stuff is going to also be broken in new ways (plus, often, the old ways too), because:
None of these OS vendors ever intends to work any product into shape such that it becomes stable, reliable, and actually what it was advertised to be when it was sold. Instead, hey, look over here, New! Shiny!
Then we have application vendors that, for no particular good reason, make their apps not just use, but depend upon new OS features. Generally speaking, you don't have to do that. You can tie a feature to an OS, and there are very good reasons to do so (the feature may not even be possible under a previous one), but then there are things that have no sane reason to be tied to an OS, such as the ability to load a new image format (Apple, I'm thinking of Aperture here.) New interface to load images through? Sure, great idea. Abandoning the old interface? Not generally a sensible thing to do. No doubt there are applications out there that use the old interface, and there will be users with (shock!) new cameras.
I find the entire cycle of abandonment to be reprehensible and ethically bankrupt. I think applications should be maintained until they aren't broken under the OS's they were designed to run under, and OS's should be maintained until they work in every way they were supposed to in the first place, and are kept as secure as possible without actually breaking things. But that's just me.
</RANT>
I've fallen off your lawn, and I can't get up.
The doomsayers were right. A great deal of effort went into patching and testing all critical systems before the year ticked over. There was no disaster because systematic action to avert it was taken well in advance.
Companies aren't "cheapskates", customers are.
Here, I'll prove my point,. You can buy something for $15 today, and have it supported until tomorrow(or whenever) or you can pay $300 for the same exact thing, only support will go for a guaranteed 10 years.
Guess what, the company didn't make the choice, you did. The company is just following the choice you've taken.
The problem is solvable. Like Cellphones, it is cheaper and easier in the long run to simply buy a new one every 2 years than it is to buy one that will last you five. And in two years, sufficient advancement means that your old cell phone won't do all the neat cool things that all the new phones want to do, and you're gonna upgrade it anyway, so buy the cheaper one now, and upgrade in two years.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
That was actually January 3982. It was easier just to let it roll over the first time round.
systemd is Roko's Basilisk.
A deadline has a wonderful way of concentrating the mind. No deadline, less motivation.
systemd is Roko's Basilisk.
Right now when someone buys a cell phone, they have it in their brains that they're making an "investment", that the phone will last for the next 20 years, or even forever.
They do? Who are these people?
For a sufficiently true portion of "everyone," "everyone" just gets a new phone every two years on contract anyway.