Slashdot Mirror
The Coming IT Nightmare of Unpatchable Systems
snydeq (1272828) writes "Insecure by design and trusted by default, embedded systems present security concerns that could prove crippling if not addressed by fabricators, vendors, and customers alike, InfoWorld reports. Routers, smart refrigerators, in-pavement traffic-monitoring systems, or crop-monitoring drones — 'the trend toward systems and devices that, once deployed, stubbornly "keep on ticking" regardless of the wishes of those who deploy them is fast becoming an IT security nightmare made real, affecting everything from mom-and-pop shops to power stations. This unpatchable hell is a problem with many fathers, from recalcitrant vendors to customers wary of — or hostile to — change. But with the number and diversity of connected endpoints expected to skyrocket in the next decade, radical measures are fast becoming necessary to ensure that today's "smart" devices and embedded systems don't haunt us for years down the line.'"
The lesson wasn't learned, but the problem was somewhat mitigated. Big software companies adopted regular patch cycles and deployed patch management tools on their customers. It kinda worked because PC are powerful computers well designed to be upgraded and modified.
This is not the case for many embedded systems. They are designed to be installed and then you forget about them. So the "classic" mitigation technique doesn't work. This is a big problem.
Well, that would be less of a problem if you didn't surf SlashDot using your refrigerator or crop-monitoring drone...
"Poorly designed", or "incorrectly designed" - perhaps. I'm fairly sure that even the ATM designers who went with an embedded MicroSoft operating system felt that they had mediated security risks adequately to deploy their systems. Incidentally, I had a chance to peek inside a local casino's slot machines - all of them, regardless of external appearance were based on an identical piece of hardware. Watching them boot showed me a MicroSoft OS underlying those slots. Not a problem, as I'm fairly certain that none of the slot machines on the floor have any conceivable way of ever connecting directly to any network except for the dark wire casinos use for exactly this purpose.
My takeaway point is that the summary is (IMHO) slightly biased. The original article appears to be well written. Just to ask - how many embedded systems should be permitted to ever connect to the internet? ATM's, for example, should demonstrably be either confined to a darknet or (as I've seen in some places) required to use dialup access. It's not perfect, but it adds a significant obstacle for crackers to overcome. The casino I mentioned earlier seems to get this point.
I don't mind smart appliances - but again, I don't see why they need internet access. The exceptions to this (smart TV's, for example) should be viewed with suspicion specifically because they are likely to be connected to the internet in some way, but my smart refrigerator probably shouldn't be - and ATM's, slot machines, SCADA systems, etc. almost certainly should never be.
Different nightmare. The Y2K embedded system nightmare was systems that wouldn't know what to do when the clock rolled over. By and large, the doomsayers were completely wrong. The current problem is *Internet enabled* embedded systems, easily hackable, out of warranty, out of support, manufacturer TU, owner/deployer isn't even sure how many they have, or where they're located, etc., etc. Picture making a botnet out of all the traffic light controllers, or the elevator controllers, or smart water meters, or internet toasters.
They are designed to be installed and then you forget about them. So the "classic" mitigation technique doesn't work. This is a big problem.
Hell, I thought the "classic" mitigation schemata for embedded devices was to not have them networked at all, leaving them to run for years (decades?) on end.
(See also the hordes of NT Telecom PBXes out there which are likely still around, requiring a goofball proprietary connection to a computer running OS/2 (!?) in order to patch it (or more commonly, you did it to add new/licensed features or to fix something gone corrupt).)
Therein lies the whole problem with the paradigm, truth be told - originally, embedded devices didn't communicate with jack shit - you unpacked it, turned it on, maybe configured it, and then you forget that it existed until it broke (at which time the vendor/contractor sent someone out to fix it), or got replaced.
All that said, hell, we already have a testbed for this nightmare - an ocean of smartphones whose carriers and manufacturers ceased to give a crap whether their wares ever got upgraded.
Quo usque tandem abutere, Nimbus, patientia nostra?
Well as an Embedded System Designer I have to speak up here, systems are usually not insecure because of lazy development, systems are insecure because clients, managers and stakeholders don't provide proper funding, deadlines or requirements. The number of times I've had to go to a manager or project manager and ask them to clarify a customers request is almost sad. The amount of times I've had to go to the same group and ask for twice or three times the amount of time to develop a solution is almost sad and the amount of times I've had to ask for much more funding to do a proper job is sad. For some unknown reason embedded designers aren't treated like normal software developers and the truth is we aren't. We don't rely on some insecure patched to hell OS to keep us safe and we don't trust laughable memory managers and kernels to keep us crash free and running smooth. We do the real work in the development world and generally it's the GUI designer who takes the credit.
We generally don't work in the world of garbage collected and managed languages, we don't work in the world where everything is already setup and ready to be called through some piss poor abstracted class implementation of system.IO and we don't get safety nets under us to catch what falls through in some kind of completely illogical and messed up exception error system ( C# ). To say embedded systems are insecure is really another way to say one of several things:
1. You didn't allocate enough time, money or proper requirements.
2. You didn't hire someone who is qualified to the job, such as putting a desktop developer onto an embedded project.
3. You didn't consider security when you dreamed up you're fragmented and broken project idea.
This is of course mitigated by a great developer who will go back to the table of executives and tell them they need what they need and won't start until it's delivered. You can't treat an embedded project like a normal software project, when you do you'll end up with systems that make Microsoft proud ( aka 0 security and patch opportunities to fly to the moon ), you need to treat an embedded project like an embedded project and give the embedded developer what he / she needs. Doing other wise will always end up you shit creek and generally the manager or stakeholder is left with the paddle looking like a fool.
<RANT>
One thing that's causing problems is the habit of Apple and Microsoft to abandon operating systems for new, often incompatible ones, instead of fixing the bugs in them. OSX 10.6.8 is full of problems; the only way to fix them is to move up to OSX 10.7 or further, which in turn can break a lot of things, because the later release isn't just fixed (if, in fact, it is fixed), it's a different animal altogether. Just one example. OS vendors take the view that you can either move forward with them, or die in a fire. Windows, Ubuntu, XP, etc... same deal.
I'm not saying these old OS's should get new features. But bugs? They should be fixed as long as humanly possible. The product was sold as having feature set X, and working. If it doesn't work as advertised, or is unreliable, it shouldn't be abandoned, it should be fixed. Except in the very rare case where it is not possible (I can't even think of one of those, actually.)
The problem is multifaceted. It isn't just that users are left with a choice of being left behind and becoming steadily more vulnerable to exploits; it is also that as the OS vendors keep jumping away from their buggy versions, the OS landscape, as it were, is left lettered with broken junk, and the new stuff is going to also be broken in new ways (plus, often, the old ways too), because:
None of these OS vendors ever intends to work any product into shape such that it becomes stable, reliable, and actually what it was advertised to be when it was sold. Instead, hey, look over here, New! Shiny!
Then we have application vendors that, for no particular good reason, make their apps not just use, but depend upon new OS features. Generally speaking, you don't have to do that. You can tie a feature to an OS, and there are very good reasons to do so (the feature may not even be possible under a previous one), but then there are things that have no sane reason to be tied to an OS, such as the ability to load a new image format (Apple, I'm thinking of Aperture here.) New interface to load images through? Sure, great idea. Abandoning the old interface? Not generally a sensible thing to do. No doubt there are applications out there that use the old interface, and there will be users with (shock!) new cameras.
I find the entire cycle of abandonment to be reprehensible and ethically bankrupt. I think applications should be maintained until they aren't broken under the OS's they were designed to run under, and OS's should be maintained until they work in every way they were supposed to in the first place, and are kept as secure as possible without actually breaking things. But that's just me.
</RANT>
I've fallen off your lawn, and I can't get up.
The doomsayers were right. A great deal of effort went into patching and testing all critical systems before the year ticked over. There was no disaster because systematic action to avert it was taken well in advance.
Companies aren't "cheapskates", customers are.
Here, I'll prove my point,. You can buy something for $15 today, and have it supported until tomorrow(or whenever) or you can pay $300 for the same exact thing, only support will go for a guaranteed 10 years.
Guess what, the company didn't make the choice, you did. The company is just following the choice you've taken.
The problem is solvable. Like Cellphones, it is cheaper and easier in the long run to simply buy a new one every 2 years than it is to buy one that will last you five. And in two years, sufficient advancement means that your old cell phone won't do all the neat cool things that all the new phones want to do, and you're gonna upgrade it anyway, so buy the cheaper one now, and upgrade in two years.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
That was actually January 3982. It was easier just to let it roll over the first time round.
systemd is Roko's Basilisk.