Slashdot Mirror

← Back to Stories (view on slashdot.org)

Now Meltdown Patches Are Making Industrial Control Systems Lurch (theregister.co.uk)

Posted by msmash on from the clock-is-ticking dept.

Patches for the Meltdown vulnerability are causing stability issues in industrial control systems. From a report: SCADA vendor Wonderware admitted that Redmond's Meltdown patch made its Historian product wobble. "Microsoft update KB4056896 (or parallel patches for other Operating System) causes instability for Wonderware Historian and the inability to access DA/OI Servers through the SMC," an advisory on Wonderware's support site explains. Rockwell Automation revealed that the same patch had caused issues with Studio 5000, FactoryTalk View SE, and RSLinx Classic (a widely used product in the manufacturing sector). "In fairness [this] may be RPC [Remote Procedure Call] change related," said cybersecurity vulnerability manager Kevin Beaumont.

16 of 98 comments (clear)

  1. Industrial systems should be super-simple by davidwr · · Score: 5, Insightful

    In general, simpler systems have a smaller attack footprint.

    Like the rest of the computer industry, many industrial systems are more complicated than they need to be.

    Yes, industrial equipment is simpler-by-design than your average general-purpose computer, but there are still some "because we can have it and it would be a nice thing to have, we have it" or "because we can buy an off-the-shelf chip that does things we don't need cheaper than paying the chip-vendor to disable unneeded functionality, we do" situations.

    There are probably innumerable industrial-control systems that can run their core functions "intelligence" on the equivalent of an early-1970s microprocessor or less. Perhaps they should.

    --
    Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
    1. Re:Industrial systems should be super-simple by gtall · · Score: 4, Insightful

      Yep, you as a control system owner can either buy (a) what's behind door number 1 that does everything you'd ever want and walks the dog when you are too tired, all for the low, low price enabled by the manufacturer selling millions, (2) what's behind door number 2 that does precisely what you want because you specified and contracted that system for your operation, all for the high, high price forced because you require a one-off.

      By the way, what's behind door number 1 comes with a volume discount so you can use it in several places in you operation. What's behind door number 2 comes with a volume discount of one because its a one-off.

      Choose wisely.

    2. Re: Industrial systems should be super-simple by guruevi · · Score: 3, Funny

      VB6 and .NET coders are cheap and the code can be cobbled together (although unreadable) with the most guaranteed billable hours and most expensive support packages ever.

      --
      Custom electronics and digital signage for your business: www.evcircuits.com
    3. Re:Industrial systems should be super-simple by davidwr · · Score: 2

      You forgot doors 3 and above:

      Door 3:

      Actual 1970s-era chips or slight revisions of them that are still being produced: Why do I need a system with an 80486 or modern-ARM-chip with a wired or wireless Ethernet Ethernet interface when a 4- or 8-bit microcontroller and a simple wired or wireless serial interface will do?

      Door 4, more expensive, feasible only if you are ordering tens or hundreds of thousands:

      Same microcontroller and same serial interface controller but with features you don't need removed during production, in much the same way that Intel disabled or removed some math functions in the i486SX for customers that didn't need them or who didn't want to pay for them (the difference being that in sub-million-unit runs you will probably pay MORE to have a function removed).

      I'm sure there are other "doors" I haven't thought of yet.

      --
      Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
    4. Re:Industrial systems should be super-simple by Rogue974 · · Score: 5, Informative

      I am a controls engineer and use the software mentioned in this post.

      First, controls guys who know anything and don't get IT telling them, you must do this now, will never install a patch until vetted by the manufacturer. I actually got a notice from the vendor saying, don't install this patch 2 days after the patch was available.

      As to being more complex then they should be or simple...

      The actual controllers that run the process are extremely simple, extremely hardened and designed to run 24/7/365. PLC processors cost $4000-$15,000 depending on type and memory and they get into the hundred of meg of memory.

      Where it gets difficult is when you start using PCs to run your operator interface. There are tons of graphics, reports, trends, etc and you use software that is designed to run on Windows, which most of your operator interfaces are designed to do.

      When a patch like this hits, the operator interface or historian has issues, but the PLC running the process keeps doing it's job, you just can't see into the PLC.

      So yes and no. There are things that are more complex and that could be simplified/run separate from windows, but those start getting prohibitively expensive and the tiny bit of extra reliability is not needed. Those kinds of systems cost 2-5 times as much and the development of those systems is more expensive because there are even fewer people with experience with it. If I had experience with those systems, I would be making 70% more then I am now and I am making enough that I don't need to complain.

    5. Re:Industrial systems should be super-simple by Rogue974 · · Score: 2

      A little off on your description. SCADA is Supervisory Controls and Data Acquisition. There are several parts and pieces to that and in most systems, that included the operator interface which is usually run on Windows machines.

      Yes, the equipment that interfaces with the field equipment is fine, but the operators can't see what that equipment is doing.

      It would be like saying, your car is fine and the engine is running, but your brakes, gas pedal, steering wheel all stopped responding and your windshield is covered in dirt so you can't see. Engine is fine though!

      I work in a facility as a controls engineer that has Wonderware and Rockwell software and I use on a daily basis the software affected by these patches. We didn't path because we don't patch until the vendors vet out patches and say it is ok and we also received the notice that said don't apply the patch.

      I know of other facilities that went down because they applied these patches. Yes, the PLCs and controllers were still working, but you can't run blind. Even if you could, the historians have the data you need for EPA compliance or to certify your product for customers so when that goes down, you stop running.

  2. VMware pulled some of their patches by El+Cubano · · Score: 2

    VMware pulled some of their patches

    Note: ESXi patches associated with VMSA-2018-0004 have been pulled down from the online and offline portal.

    ...

    For ESXi hosts that have not yet applied one of the following patches ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG, VMware recommends not doing so at this time. It is recommended to apply the patches listed in VMSA-2018-0002 instead.

    ...

    For servers using the Intel Haswell and Broadwell processors (see Table 1 for the specific list of affected VMware vSphere supported Intel Haswell and Broadwell processors) that have applied ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG VMware recommends the following:

    ...

    VMware is working closely with Intel and the industry to come to a quick resolution of this Intel microcode issue and provide an update to our customers as soon as possible.

    Oops!

    reference

    1. Re:VMware pulled some of their patches by El+Cubano · · Score: 3, Insightful

      I guess I should have finished my thought.

      It's not just industrial control systems, but hypervisors, and plain old systems too. It sees like this is an object lesson in how speed (in terms of releasing a fix) comes at a cost of performance/quality. I know people were all in a panic once Meltdown and Spectre became public, but this wasn't just fixing a SQL injection vulnerability in Rails or Django. This fundamentally affected the execution of nearly every instruction to go through affected CPUs.

      I suspect that the severity and publicity made a more organized roll out with extensive beta testing impossible for just about every vendor that had affected products.

  3. Stuxnet is now crashing by Anonymous Coward · · Score: 3, Insightful

    Now things like Stuxnet won't be able to infiltrate as easily. WTF are these things doing connected anyway, and if not connected why do they need the patches? And don't get me started on Windows...

  4. Toldja so... by GerryGilmore · · Score: 5, Insightful

    From the very beginning, I've tried to get everyone to pause the Panic Parade, but nnnnnooooooo. To try to address probably the most complex vulnerability yet discovered (it took over 20 YEARS for this to be found) that also requires you to already be running malware on your system, people are flashing new BIOSes, patching kernels and generally behaving like idiots. Slow FT down, folks! Let the CPU and OS experts have a real shot at minimizing the risk, without killing our production systems, FFS!!

    1. Re:Toldja so... by edtice1559 · · Score: 2

      I guess this has somewhat of a political slant, but I don't think a -1 moderation is really fair. The reality is that much of society transitioned from healthy skepticism to assuming that experts are always wrong. That's unfortunate because we need experts *especially* in situations where the experts are most likely to be wrong. Even when experts are "wrong" they usually provide advice that is reasonable enough that we can recover when new information emerges. But the fact is that being unqualified is suddenly a qualification.

    2. Re:Toldja so... by duke_cheetah2003 · · Score: 2

      Yeah, pretty much everything GerryGilmore said. This knee-jerk reaction to a pretty obscure flaw is way overboard.

      I personally don't want my CPU's branch prediction gimped because some other idiot can't keep his web browser away from malicious sites.

      The only panic that should be realistic and warranted is big cloud VM providers concerned these attacks could compromise tenants on shared systems. Patch that shit into oblivion as far as I'm concerned, but get your grubby patches off my desktop. I don't want it, I'll take the performance any time of the day, I know how to avoid questionable web sites and content.

  5. Also Beckhoff TwinCAT 3 by RobinH · · Score: 4, Interesting

    We received a notification from Beckhoff to avoid these patches for TwinCAT 3 until they would patch their runtime to be compatible. We update through WSUS so we were able to do that. Beckhoff themselves urge you *not* to install Windows Updates on their control system PCs even though they bill their product as part of the "Internet of Things" and play up the connectivity of everything. They're hypocrites, but Rockwell did the same thing when we used their product. They wouldn't warranty their software if you installed anti-virus on the same server as their historian product.

    --
    "I have never let my schooling interfere with my education." - Mark Twain
  6. Industrial systems don't have as much spare room by plague911 · · Score: 4, Insightful

    I have never worked on industrial systems, I did work on some large scale defense equipment. One of the design considerations is cost, in order to minimize cost, you match the components spec to the semi-well defined performance need. No need on buying a V12 when a V6 will do...... Now I am not saying you don't build in some buffer, but the MASSIVE performance hit required by these patches could easily blow the given performance buffer out of the water. I could easily see how billions of dollars worth of industrial systems simply will not be able to patched due performance cost of the patches. Additionally given the age/design of the systems there is no way to conveniently upgrade the systems.

  7. The hazards of monoculture by plopez · · Score: 3, Insightful

    relying on one piece of tech is as bad as relying on one food crop.

    --
    putting the 'B' in LGBTQ+
  8. No they shouldn't, simplicity is too complicated by Anonymous Coward · · Score: 2

    an 8-bit micro-controller won't cut it anymore.

    Everything is so precision motion-based and customizable.
    Everything needs to have comprehensive diagnostics to say what's wrong (because troubleshooting has been replaced with magical thinking).
    Everything is running at multiples of 125 microseconds, now even update rates of 10 milliseconds is too long.

    Serial (RS232) is an absolute nightmare to diagnose. RS485 is a nightmare to get them to wire it correctly. YES, RS485 as in 2 wires and a shield is still an absolute troubleshooting nightmare. Colorblind landing wires wrong, cutting off the shielding, not piercing insulation, exceeding bend radius, signal reflections, bad splices, crushed or overflexed cable... Industrial Ethernet wins hands down for its ability for diagnostic troubleshooting, because what's electrically simple is horrendously complicated when bureaucracies and ignoramus is involved.

    Simple isn't simple. You can stuff a 32-bit CPU with megs of RAM into the same simple looking box as a vintage 8051 board. One that nobody is going to bother to desolder, let alone even take apart and diagnose at the component level when it fails.