Now Meltdown Patches Are Making Industrial Control Systems Lurch (theregister.co.uk)

Patches for the Meltdown vulnerability are causing stability issues in industrial control systems. From a report: SCADA vendor Wonderware admitted that Redmond's Meltdown patch made its Historian product wobble. "Microsoft update KB4056896 (or parallel patches for other Operating System) causes instability for Wonderware Historian and the inability to access DA/OI Servers through the SMC," an advisory on Wonderware's support site explains. Rockwell Automation revealed that the same patch had caused issues with Studio 5000, FactoryTalk View SE, and RSLinx Classic (a widely used product in the manufacturing sector). "In fairness [this] may be RPC [Remote Procedure Call] change related," said cybersecurity vulnerability manager Kevin Beaumont.

Comments

Industrial systems should be super-simple

[davidwr]

In general, simpler systems have a smaller attack footprint.

Like the rest of the computer industry, many industrial systems are more complicated than they need to be.

Yes, industrial equipment is simpler-by-design than your average general-purpose computer, but there are still some "because we can have it and it would be a nice thing to have, we have it" or "because we can buy an off-the-shelf chip that does things we don't need cheaper than paying the chip-vendor to disable unneeded functionality, we do" situations.

There are probably innumerable industrial-control systems that can run their core functions "intelligence" on the equivalent of an early-1970s microprocessor or less. Perhaps they should.

Re:Industrial systems should be super-simple

[gtall]

Yep, you as a control system owner can either buy (a) what's behind door number 1 that does everything you'd ever want and walks the dog when you are too tired, all for the low, low price enabled by the manufacturer selling millions, (2) what's behind door number 2 that does precisely what you want because you specified and contracted that system for your operation, all for the high, high price forced because you require a one-off.

By the way, what's behind door number 1 comes with a volume discount so you can use it in several places in you operation. What's behind door number 2 comes with a volume discount of one because its a one-off.

Choose wisely.

Re: Industrial systems should be super-simple

[guruevi]

VB6 and .NET coders are cheap and the code can be cobbled together (although unreadable) with the most guaranteed billable hours and most expensive support packages ever.

Re:Industrial systems should be super-simple

[Rogue974]

I am a controls engineer and use the software mentioned in this post.

First, controls guys who know anything and don't get IT telling them, you must do this now, will never install a patch until vetted by the manufacturer. I actually got a notice from the vendor saying, don't install this patch 2 days after the patch was available.

As to being more complex then they should be or simple...

The actual controllers that run the process are extremely simple, extremely hardened and designed to run 24/7/365. PLC processors cost $4000-$15,000 depending on type and memory and they get into the hundred of meg of memory.

Where it gets difficult is when you start using PCs to run your operator interface. There are tons of graphics, reports, trends, etc and you use software that is designed to run on Windows, which most of your operator interfaces are designed to do.

When a patch like this hits, the operator interface or historian has issues, but the PLC running the process keeps doing it's job, you just can't see into the PLC.

So yes and no. There are things that are more complex and that could be simplified/run separate from windows, but those start getting prohibitively expensive and the tiny bit of extra reliability is not needed. Those kinds of systems cost 2-5 times as much and the development of those systems is more expensive because there are even fewer people with experience with it. If I had experience with those systems, I would be making 70% more then I am now and I am making enough that I don't need to complain.

Stuxnet is now crashing

[Anonymous Coward]

Now things like Stuxnet won't be able to infiltrate as easily. WTF are these things doing connected anyway, and if not connected why do they need the patches? And don't get me started on Windows...

Re:VMware pulled some of their patches

[El Cubano]

I guess I should have finished my thought.

It's not just industrial control systems, but hypervisors, and plain old systems too. It sees like this is an object lesson in how speed (in terms of releasing a fix) comes at a cost of performance/quality. I know people were all in a panic once Meltdown and Spectre became public, but this wasn't just fixing a SQL injection vulnerability in Rails or Django. This fundamentally affected the execution of nearly every instruction to go through affected CPUs.

I suspect that the severity and publicity made a more organized roll out with extensive beta testing impossible for just about every vendor that had affected products.

Toldja so...

[GerryGilmore]

From the very beginning, I've tried to get everyone to pause the Panic Parade, but nnnnnooooooo. To try to address probably the most complex vulnerability yet discovered (it took over 20 YEARS for this to be found) that also requires you to already be running malware on your system, people are flashing new BIOSes, patching kernels and generally behaving like idiots. Slow FT down, folks! Let the CPU and OS experts have a real shot at minimizing the risk, without killing our production systems, FFS!!

Also Beckhoff TwinCAT 3

[RobinH]

We received a notification from Beckhoff to avoid these patches for TwinCAT 3 until they would patch their runtime to be compatible. We update through WSUS so we were able to do that. Beckhoff themselves urge you *not* to install Windows Updates on their control system PCs even though they bill their product as part of the "Internet of Things" and play up the connectivity of everything. They're hypocrites, but Rockwell did the same thing when we used their product. They wouldn't warranty their software if you installed anti-virus on the same server as their historian product.

Industrial systems don't have as much spare room

[plague911]

I have never worked on industrial systems, I did work on some large scale defense equipment. One of the design considerations is cost, in order to minimize cost, you match the components spec to the semi-well defined performance need. No need on buying a V12 when a V6 will do...... Now I am not saying you don't build in some buffer, but the MASSIVE performance hit required by these patches could easily blow the given performance buffer out of the water. I could easily see how billions of dollars worth of industrial systems simply will not be able to patched due performance cost of the patches. Additionally given the age/design of the systems there is no way to conveniently upgrade the systems.

The hazards of monoculture

[plopez]

relying on one piece of tech is as bad as relying on one food crop.