Slashdot Mirror
The Coming IT Nightmare of Unpatchable Systems
snydeq (1272828) writes "Insecure by design and trusted by default, embedded systems present security concerns that could prove crippling if not addressed by fabricators, vendors, and customers alike, InfoWorld reports. Routers, smart refrigerators, in-pavement traffic-monitoring systems, or crop-monitoring drones — 'the trend toward systems and devices that, once deployed, stubbornly "keep on ticking" regardless of the wishes of those who deploy them is fast becoming an IT security nightmare made real, affecting everything from mom-and-pop shops to power stations. This unpatchable hell is a problem with many fathers, from recalcitrant vendors to customers wary of — or hostile to — change. But with the number and diversity of connected endpoints expected to skyrocket in the next decade, radical measures are fast becoming necessary to ensure that today's "smart" devices and embedded systems don't haunt us for years down the line.'"
They had the same problem prior to the year 2000, so why wasn't this lesson already learned?
Feed the need: Digitaladdiction.net
Slashdot keeps forwarding me to activeplayer.us, which tries to drive-by-download an installer. It looks like Adobe's FlashPlayer site.
Support my political activism on Patreon.
Wait until we have driverless cars on the road. But I'm sure they'll all be bullet-proof secure, don'tcha think?
"Poorly designed", or "incorrectly designed" - perhaps. I'm fairly sure that even the ATM designers who went with an embedded MicroSoft operating system felt that they had mediated security risks adequately to deploy their systems. Incidentally, I had a chance to peek inside a local casino's slot machines - all of them, regardless of external appearance were based on an identical piece of hardware. Watching them boot showed me a MicroSoft OS underlying those slots. Not a problem, as I'm fairly certain that none of the slot machines on the floor have any conceivable way of ever connecting directly to any network except for the dark wire casinos use for exactly this purpose.
My takeaway point is that the summary is (IMHO) slightly biased. The original article appears to be well written. Just to ask - how many embedded systems should be permitted to ever connect to the internet? ATM's, for example, should demonstrably be either confined to a darknet or (as I've seen in some places) required to use dialup access. It's not perfect, but it adds a significant obstacle for crackers to overcome. The casino I mentioned earlier seems to get this point.
I don't mind smart appliances - but again, I don't see why they need internet access. The exceptions to this (smart TV's, for example) should be viewed with suspicion specifically because they are likely to be connected to the internet in some way, but my smart refrigerator probably shouldn't be - and ATM's, slot machines, SCADA systems, etc. almost certainly should never be.
"Unpatchable" does not mean "Unsecured" in fact, I'd say it adds to security in many senses. A system that can't be patched, can also not be altered to do the attackers bidding. At the very least, any privileges the attacker may have access to can not be elevated to create some even worse situation. Worst case scenario you just disconnect power to the device in question. Submit it for warranty repair. If you're using a closed source software product out of warranty/support it's your own stupid fault.
Explain to me where I'm supposed to get my GEOS updated for my Commodore 64? And I sent in my warranty registration card, but I never heard back from them.
There are two bleeding edges. One is the leading edge of cutting technology.
There other is the trailing edge where systems age out because they take a lot of effort to update.
One way the trailing edge can not be updated because the overall system is designed to where there are critical parts that can not be monkeyed with in a low risk scenario. (This does happen).
The other option on the trailing edge is where the systems are not worth the effort. Most of the Internet of Everything appliances really have zero income after the first few months and yet are expected to have a longer lifetime than many major IT infrastructure requirements.
And - yes, these kind of incidents are mistakes. I stand by my previous assertion that nobody set out to create insecure embedded systems. Poor design, incorrect design, or just plain inept management oversight has led to these kinds of mistake. Much as I'd like to blame MicroSoft for all of it, I can't. Sorry - love to, can't. I'm still certain that all of the entities involved believed they had correctly and adequately mediated the risks . . . that, or they had some PHB breathing down their necks to do as they were told. Happens all the time - ask Scott Adams.
Your fridge sends out a little packet that says: "Hey, I am past my warranty! Time to up the ad volume to MAX". Or: "Please press OK to agree to the new privacy terms and conditions or your device wont work anymore".
There are many serious problems that are here NOW and must be addressed.
I am more worried about people not patching what can be patched.
Make them patchable over the internet by default.
Oh, wait...
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
The overall level of system quality is so piss poor anyway what does it matter than your toaster is going to try to kill Sarah Connor? Anyone read the news recently? Car makers recalled about eleventy zillion cars recently and half the problems were on board computer based. Are you going to lose any sleep that your refrigerator will get hacked and join Skynet? Because the real problem is going to be that when your Refrigerator blows an 80 cent part on a 2 dollar circuit board it's going to cost $1100 to 'repair'.
Probably not unless the user wants it fixed, and most don't. People have plenty of experiences with patches breaking new things, or taking away old functionality they had come to depend on. When someone tells me "this patch will solve all your problems", they usually aren't advertising the list of new problems they're creating for me. Anyone who plays iPhone app games knows that the patches sometimes come with game-stopping bugs; other patches have been known to suddenly add annoying advertising.
Usually, I'm at a point of equilibrium where I am at least coping with the bugs in the devices surrounding me. If I know that the "mute button" on my GoogleTV box doesn't work unless I press it twice, I simply learn to press it twice; while I know it's a stupid workaround, it's one I can live with. What I might not be able to live with are the bugs that come with the next round of patches.
Now, we make that experience hurdle even harder to scale: as a end user, I think security patches are worse than regular patches. The end user doesn't see a physical benefit from the patches, but knows he might suffer. What does he care if his thermostat or washing machine is sending spam around the world, as long as his house is warm and his clothes are clean? But if he installs the patches, he risks having a cold house or dirty clothes, or even advertisements streaming across his refrigerator's screen. It's just not worth the risk to patch them.
And if you want to see a really risk-averse, don't-patch-me crowd, talk to the SCADA industrial control people. If you suggest you need to update the software running the sewage ejection pump, the city engineer is going to hand you an invoice for $20,000 and say "that covers my cost of testing your patch."
John
So you get a couple of years to use a product that is riddled with security holes which aren't going to be patched, and then you need to buy another buggy product for another round of exploits that you can't defend against because there still aren't going to be any updates. Just because a product gets an official end-of-life date doesn't mean it magically becomes secure until that date.
The software industry needs to made liable for software flaws. Way too many businesses write shoddy software for applications with a massive security footprint, always chasing first-to-market. There isn't just no incentive to get it right, there is a huge incentive to get it wrong by riding roughshod over code quality in order to save time. The situation won't improve until releasing products with exploitable software creates a dire financial risk for the manufacturer.
The problem IS that things are trusted by default... but not in the way the author thought. If you trust every program you run by default, you are doomed. An operating system should NEVER trust anything by default... Linux, Windows, OSX all violate this principle. So do embedded devices base on some variant of them.
Never trust by default, and you stop having to worry about side-effects, and start deciding what the limits are ahead of time.
That nightmare is a reality for most Sys/Security Admins. Very common are control systems that you simply cannot touch, or end up out of compliance with the vendor and unsupported. Companies with multi million dollar control systems don't want to hear about "patches" and "vulnerabilities", they just need them to work for business productivities sake. But of course the vendor needs remote access, so the device needs internet access.
There is no coming nightmare, its been a part of life for decades, insecure systems with business needs and no concern for security implications. The business accepts the risk despite protests and it gets setup or you get fired.
Those are maggots, not Rice-A-Roni.
Well as an Embedded System Designer I have to speak up here, systems are usually not insecure because of lazy development, systems are insecure because clients, managers and stakeholders don't provide proper funding, deadlines or requirements. The number of times I've had to go to a manager or project manager and ask them to clarify a customers request is almost sad. The amount of times I've had to go to the same group and ask for twice or three times the amount of time to develop a solution is almost sad and the amount of times I've had to ask for much more funding to do a proper job is sad. For some unknown reason embedded designers aren't treated like normal software developers and the truth is we aren't. We don't rely on some insecure patched to hell OS to keep us safe and we don't trust laughable memory managers and kernels to keep us crash free and running smooth. We do the real work in the development world and generally it's the GUI designer who takes the credit.
We generally don't work in the world of garbage collected and managed languages, we don't work in the world where everything is already setup and ready to be called through some piss poor abstracted class implementation of system.IO and we don't get safety nets under us to catch what falls through in some kind of completely illogical and messed up exception error system ( C# ). To say embedded systems are insecure is really another way to say one of several things:
1. You didn't allocate enough time, money or proper requirements.
2. You didn't hire someone who is qualified to the job, such as putting a desktop developer onto an embedded project.
3. You didn't consider security when you dreamed up you're fragmented and broken project idea.
This is of course mitigated by a great developer who will go back to the table of executives and tell them they need what they need and won't start until it's delivered. You can't treat an embedded project like a normal software project, when you do you'll end up with systems that make Microsoft proud ( aka 0 security and patch opportunities to fly to the moon ), you need to treat an embedded project like an embedded project and give the embedded developer what he / she needs. Doing other wise will always end up you shit creek and generally the manager or stakeholder is left with the paddle looking like a fool.
<RANT>
One thing that's causing problems is the habit of Apple and Microsoft to abandon operating systems for new, often incompatible ones, instead of fixing the bugs in them. OSX 10.6.8 is full of problems; the only way to fix them is to move up to OSX 10.7 or further, which in turn can break a lot of things, because the later release isn't just fixed (if, in fact, it is fixed), it's a different animal altogether. Just one example. OS vendors take the view that you can either move forward with them, or die in a fire. Windows, Ubuntu, XP, etc... same deal.
I'm not saying these old OS's should get new features. But bugs? They should be fixed as long as humanly possible. The product was sold as having feature set X, and working. If it doesn't work as advertised, or is unreliable, it shouldn't be abandoned, it should be fixed. Except in the very rare case where it is not possible (I can't even think of one of those, actually.)
The problem is multifaceted. It isn't just that users are left with a choice of being left behind and becoming steadily more vulnerable to exploits; it is also that as the OS vendors keep jumping away from their buggy versions, the OS landscape, as it were, is left lettered with broken junk, and the new stuff is going to also be broken in new ways (plus, often, the old ways too), because:
None of these OS vendors ever intends to work any product into shape such that it becomes stable, reliable, and actually what it was advertised to be when it was sold. Instead, hey, look over here, New! Shiny!
Then we have application vendors that, for no particular good reason, make their apps not just use, but depend upon new OS features. Generally speaking, you don't have to do that. You can tie a feature to an OS, and there are very good reasons to do so (the feature may not even be possible under a previous one), but then there are things that have no sane reason to be tied to an OS, such as the ability to load a new image format (Apple, I'm thinking of Aperture here.) New interface to load images through? Sure, great idea. Abandoning the old interface? Not generally a sensible thing to do. No doubt there are applications out there that use the old interface, and there will be users with (shock!) new cameras.
I find the entire cycle of abandonment to be reprehensible and ethically bankrupt. I think applications should be maintained until they aren't broken under the OS's they were designed to run under, and OS's should be maintained until they work in every way they were supposed to in the first place, and are kept as secure as possible without actually breaking things. But that's just me.
</RANT>
I've fallen off your lawn, and I can't get up.
[...]affecting everything from mom-and-pop shops to power stations. This unpatchable hell is a problem with many fathers, from recalcitrant vendors to customers wary of[...]
This is a weighty issue. I will take it before the elders of my own company -- surely those wise fathers will know what to do. In the meantime, send forth the maidens to wail and weep in the streets, that all the people may know how grievous is this news.
It's Bladerunner all over again.
What if that could save you money? (it can.) What if it adds convenience and security? (it can.) What if it informs you about your usage such that you can improve your comfort level? (it can.) What if it gives you remote information, such as "the heater has failed, the pipes will freeze, you need to come deal with this" (it can.) What then? Still no business being Internet enabled?
It's not a failure of needlessly Internetting the device; it's a failure of vision on your part (and perhaps a failure on the manufacturer's part to make a secure device... that can be fixed, and pressure should be applied so the fix happens.) Sure, you can get along with your old thermostat. You could get along with a coal stove instead of a gas or electric range, too. But most of the time, not such a good idea.
The problem isn't accessibility. That's just a stopgap, though certainly a highly effective one. The real problem is security. Worthy of raving about, for sure. But with the idea of making it actually secure -- not of dumping capability out the window because of too little effort expended.
I've fallen off your lawn, and I can't get up.
We don't have unpatchable systems. What we have are vendors not wanting to maintain support for too long as they want to force people to buying always the newest to generate revenue.
There is this overall trend in IT industry that hardware gets softer and softer. With every generation, more features are implemented in software, and therefore are, in theory, patchable. But the possibilities of the soft hardware don't meet the commercial interest of the companies.
We have multiple benefits when using computer machines for doing human's work. But we also need to realize this doesn't come for free. Either we live with vulnerable systems, or we update them, simple as that. When purchasing new hardware it should always be a question to ask whether the software can be updated, and how the hw will be maintained. Compliances usually have a bad performance in this. Use well known parts, and be as mainstream as possible.
Computers don't have a long history of serving humans yet. I hope these update issues are a problem of the first generations.
Why would I ever want my refrigerator to have internet access?
The marketers, sure, they want me to think that I need that. But, really, what conceivable value or advantage would the ($30-extra purchase price) confer to me?
None? Well, I must be a sucker.
Or, wait, I have to actually exert more effort to maintain the internet security of my refrigerator, which wasn't and should have never been internet-connected in the first place? If you find yourself in this latter situation, you are dumber than a sucker, mark, or rube. You are the problem.
I think we have to face the fact that we're moving beyond an era where we can secure systems and instead need to move towards mitigating the damage.
Let's think about our unupgradeable internet enabled toaster that counts our calories and orders fresh bread when it detects we've used up what we have. If that toaster gets hacked there are a few possible results:
1) It might set your house on fire. This should be mitigated by all toasters having appropriate physical sensors that are not software controlled to prevent a fire. A simple thermal fuse would cost only a cent or two. A manufacturer who builds a toaster that can be set on fire over the internet under any circumstances should face significant liability.
2) Your toaster might be turned into a spam machine or bitcoin miner or something similar. If this renders your toaster non-functional then you will throw it out because its broken and its no longer a problem.
3) Your toaster might be more carefully owned and remain functional. This is obviously the worst case. But the way to handle this is with improved perimiter defenses Routers should be enhanced to monitor for suspicious activity. You could get a virus alert or similar that notifies you your toaster is behaving oddly.
The level of protection needed depends on the device. Something with a camera or microphone needs more thoughtful security than a toaster. (Until our toasters include facial recognition to tune the desired level of toastiness).
Another related thought. One big issue we have is embedded systems are often networked together. Traffic lights for instance. My first choice would be that such devices not be on the internet, but if they must I think we could create some isolation or sandboxing. Imaging if each embedded traffic light had a mini-router chip that had some sort of unalterable channel code. Make sure that a traffic light can only talk to other traffic lights or control hardware with the same channel code. Beyond that, I think you are going to again have to rely on perimiter defenses built into routers to detect and interdict command/control from hackers and detect abuse of the traffic lights. Networked but safety critical systems such as traffic lights should have a fallback unnetworked mode (old fashioned timing in the case of traffic lights).
The point is there isn't any one size fits all solution but if we focus on risk reduction, periphery detection and, where critical, ways to disable networked behavior we can protect our infrastructure significantly better than it is now.
player life has a web site and is tied to games in lot's of casinos
You may not want to connect your thermostat to the Internet, but you may want it connected to your home network, which so happens to have Internet access. The heating and cooling system in my old home for 10 years ago kept a multi-year log of each and every time the heating or cooling kicked on, what temp it was, what the humidity was, and all kinds of other stuff. Accessing it over a serial port was annoying. It would have been a lot more convenient if it had a web server that ran over wifi or Ethernet.
You know, all of this stuff MUST be connected to the internet.. or it will EXPLODE!
Oh wait it wont.. so just not plugging it in makes it 100% hack proof.
Do not look at laser with remaining good eye.
Microsoft doesn't want to produce a new version of Windows; they want to make money and selling new releases of Windows is how they accomplish this.
I truly do not understand why they are nixing Windows XP. The money making opportunity is tremendous: Take 1/10th of their O/S development team, and have them work on bug fixes for Windows XP. Pay them by charging subscriptions for XP support. It wouldn't have to be much: maybe $10 to $20 per year would be more than enough, and those still hanging on to Windows XP would very likely be completely happy to pay $12.95 per year to to have to mess with their obviously working system.
If you assume that the $25 or so that MS gets for a OS license from vendors covers 3 years, then $12.95 per year is at least 100% to 200% more per year, from customers that don't demand or want new features. It's like shooting fish in a barrel! If it's true that 30% of computers are still running Windows XP, this would easily become one of the most profitable divisions of MS in the near future.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
I have an Onkyo amplifier (mid-range) and an LG BlueRay player (low-end). A few months back, the Onkyo no longer could connect to Rhapsody (yah, I know, Rhapsody, but the wife likes it). Onkyo knows about it, and basically says "tough" because it's an old model (~ 4 years). I can use Chromecast, but it's an annoyance, because now I have to have a phone or tablet around to listen to music. The BlueRay player no longer shows images for Netflix in its bundled application. I can use Chromecast, but again, it's annoying. It's apparently in neither company's interest to update the firmware (which is updateable on both devices) to fix these issues, because they believe I will go out and by a more recent device (if I do, obviously it will be from neither of these companies).
The whole concept of integrated A/V appliances continues to underwhelm me. Fortunately, I didn't drop extra coin for a "smart" TV, but it seems that it's how the market is moving.
Well said! Now if we could only get people also to apply the same ideas to fixing up programming languages and libraries (e..g Swift vs. C/D/Java/JavaScript/Smalltalk/etc.) instead of inventing new ones that just have different gaps and different bugs...
A 21st century issue: the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.
Companies aren't "cheapskates", customers are.
Here, I'll prove my point,. You can buy something for $15 today, and have it supported until tomorrow(or whenever) or you can pay $300 for the same exact thing, only support will go for a guaranteed 10 years.
And here is a counterpoint: I was evaluating a piece of robust hardware for installation at remote sites (~$5k). The hardware has a built in micro that monitors all the functions and provides configuration, it is programmed via DIP switches and a serial port, and output status on LEDs and relays (good). The company offers a $700 "TCP/IP" option that provides SNMP monitoring and configuration over IP, as well as uploads all the site info "to the cloud"... because that is all the rage these days.
... The $700 option is a rebadged BeagleBoard connected to the serial port. Do you really think this is going to be supported more than a year or two?
Beta sucks... and completely screwed the above quoting and formating....
"Computers don't have a long history of serving humans yet."
http://en.wikipedia.org/wiki/T...
Or the recent Slashdot article on robots being used to rip apart mosquitoes...
http://science.slashdot.org/st...
Or previously, slugs:
http://science.slashdot.org/st...
""SlugBot is no ordinary robot. SlugBot hunts down slugs, and is powered by fermenting the slugs' corpses, producing biogas fuel. "
See also, for a different robotic dystopia from helping too much and "protecting" too much: http://en.wikipedia.org/wiki/W...
Life seems livvd between fire and ice, between order and chaos.
Computers and software have been through several generations over the last 70 years (and more). The failures we still see IMHO have more to do with our social systems (including legal frameworks) than with the possibilities of hardware or software. The same is perhaps true of nuclear power -- Fukushima happened more for social reasons than technical ones..
A 21st century issue: the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.
6 months after the whole issue of embedded systems blew up in Mom and Pop's pizza shop router is breaking news for InfoWorld.
I don't want to think about the number of times more visionary people have brought up this very topic over the past 15 years.
I wonder I'll be concerned in 20+ years - after I've retired from my career and will be paying rent on my hot-bunk from earnings I make washing dogs.
I'm not a software engineer, just a user, but why do we need to have new OS versions every fucking year? Really, couldn't Microsoft have an OS that looks largely like Windows 3.1 or XP or whatever to the user, but is streamlined and patched to modern standards of security and interoperability?
I'm not trying to start a flame war, but really - as a user I see more change in software as churning to turn a dollar vs. actual improvement. A model where a software might be patched and "recalled" for improvement for a long period of time would be much more satisfactory than the jiggles in UI format that seem to introduce more bugs than improvements.
And yeah, this costs money to do. What cost <X> would I have to plunk down *once* to sit and enjoy an stable OS that were patched and upgraded ad infinitum (or until I died, close enough)?
This must-churn-a-new-shiny-every-reporting-quarter is bullshit. I need a stable OS that I can operate for years without relearning everything. Is such a business model even possible?
Left MS Windows for Linux Mint and never looked back!
Vote for Bernie in 2016!
No? What if it prints darth vader on the toast for your kids, but they want han solo? What if the fuzzy logic that makes sure the toast is properly browned doesn't work on darker bread, but they figure it out and can upgrade it and the wife LOVES darker breads? What if it prints JAR JAR on the toast but you could upgrade it to print Leia??? JAR JAR man, you HAVE to get rid of that, it'll crush your kid's very SOULS.
I've fallen off your lawn, and I can't get up.
Software has made people complacent with regard to code quality. You can always patch, can't you? Well, you can't. Get it right!
No. The Internet made them complacent; before that, patches were a major task, as the end user had to download them from a bulletin board somewhere, or an engineer might have to fly half way around the world to install it for you.
But the same Internet that has made patching easy has also made the lack of patches a remotely-exploitable route for hacking into random crap that should never have been on it in the first place.
It's funny you think people patch their systems _now_
I will shred my adversaries. Pull their eyes out just enough to turn them towards their mewing, mutilated faces. Illyria
What do they mean "the coming nightmare..." If you've worked with Industrial or smaller IT customers, it's already here. I just had to deal with a CNC machine that, no kidding, ran OS2 Warp version 3! How do you get patches for that?