Now Meltdown Patches Are Making Industrial Control Systems Lurch (theregister.co.uk)

Patches for the Meltdown vulnerability are causing stability issues in industrial control systems. From a report: SCADA vendor Wonderware admitted that Redmond's Meltdown patch made its Historian product wobble. "Microsoft update KB4056896 (or parallel patches for other Operating System) causes instability for Wonderware Historian and the inability to access DA/OI Servers through the SMC," an advisory on Wonderware's support site explains. Rockwell Automation revealed that the same patch had caused issues with Studio 5000, FactoryTalk View SE, and RSLinx Classic (a widely used product in the manufacturing sector). "In fairness [this] may be RPC [Remote Procedure Call] change related," said cybersecurity vulnerability manager Kevin Beaumont.

Comments

Industrial systems should be super-simple

[davidwr]

In general, simpler systems have a smaller attack footprint.

Like the rest of the computer industry, many industrial systems are more complicated than they need to be.

Yes, industrial equipment is simpler-by-design than your average general-purpose computer, but there are still some "because we can have it and it would be a nice thing to have, we have it" or "because we can buy an off-the-shelf chip that does things we don't need cheaper than paying the chip-vendor to disable unneeded functionality, we do" situations.

There are probably innumerable industrial-control systems that can run their core functions "intelligence" on the equivalent of an early-1970s microprocessor or less. Perhaps they should.

Re:Industrial systems should be super-simple

[Anonymous Coward]

No, they should just use standard Ubuntu/Debian/CentOS or something because then at least you can update the thing, and just implement their own thing in python so that you can easily read the code and inspect what it is doing.

Re:Industrial systems should be super-simple

[gtall]

Yep, you as a control system owner can either buy (a) what's behind door number 1 that does everything you'd ever want and walks the dog when you are too tired, all for the low, low price enabled by the manufacturer selling millions, (2) what's behind door number 2 that does precisely what you want because you specified and contracted that system for your operation, all for the high, high price forced because you require a one-off.

By the way, what's behind door number 1 comes with a volume discount so you can use it in several places in you operation. What's behind door number 2 comes with a volume discount of one because its a one-off.

Choose wisely.

Re:Industrial systems should be super-simple

[MatthiasF]

I agree, the problem is since most developers use Windows they will design their applications on it.

If we can get more developers in the industrial sectors to swap to something more cross-platform then they could use locked down ARM devices like Raspberry Pi's that can be swapped out more easily to check for security audits.

Re: Industrial systems should be super-simple

[guruevi]

VB6 and .NET coders are cheap and the code can be cobbled together (although unreadable) with the most guaranteed billable hours and most expensive support packages ever.

Re:Industrial systems should be super-simple

[davidwr]

You forgot doors 3 and above:

Door 3:

Actual 1970s-era chips or slight revisions of them that are still being produced: Why do I need a system with an 80486 or modern-ARM-chip with a wired or wireless Ethernet Ethernet interface when a 4- or 8-bit microcontroller and a simple wired or wireless serial interface will do?

Door 4, more expensive, feasible only if you are ordering tens or hundreds of thousands:

Same microcontroller and same serial interface controller but with features you don't need removed during production, in much the same way that Intel disabled or removed some math functions in the i486SX for customers that didn't need them or who didn't want to pay for them (the difference being that in sub-million-unit runs you will probably pay MORE to have a function removed).

I'm sure there are other "doors" I haven't thought of yet.

Re:Industrial systems should be super-simple

[Anonymous Coward]

The systems affected are not the actual PLCs that directly control the equipment.

These are the HMIs and programming software that runs on a windows workstation.
(The Panel-Mounted HMIs also run typically WinCE, although the newer PV5500s run linux)
And those are *anything but simple*.

I run all my industrial programming software in VMs for just that reason--there are bizarre dependencies and incompatibilities between different manufacturers of this software, and even between different vintages of software from the same mfr.

The rigor with which the embedded software in the PLCs (that actually control the equipment) are tested and proven is fairly high.
For the PC-based development and visualization software, it's somewhat less rigorous.

Just as a qualifier, I've been programming with this stuff--Allen Bradley (now Rockwell), Siemens, Mitsubishi, GE/Fanuc, Omron, Modicon, PLC-Direct, etc. etc. etc. for about 26 years. And testing it for one MFR off and on for about 21 years.
The PLCs are pretty damned solid.
The PC Based stuff (HMIs, Programming SW) is not expected to be nearly as robust, and that expectation is vigorously satisfied.

(Stuxnet, for example got in on the Visualization "PC" side)

Re:Industrial systems should be super-simple

[tlhIngan]

The problem is actually not the SCADA system itself. It's the computer monitoring the SCADA equipment!

The meltdown and spectre patches that Microsoft released are apparently causing problems with the monitoring and configuration software that runs on the computers attached to the SCADA network. The equipment is fine.

Re:Industrial systems should be super-simple

[Rogue974]

I am a controls engineer and use the software mentioned in this post.

First, controls guys who know anything and don't get IT telling them, you must do this now, will never install a patch until vetted by the manufacturer. I actually got a notice from the vendor saying, don't install this patch 2 days after the patch was available.

As to being more complex then they should be or simple...

The actual controllers that run the process are extremely simple, extremely hardened and designed to run 24/7/365. PLC processors cost $4000-$15,000 depending on type and memory and they get into the hundred of meg of memory.

Where it gets difficult is when you start using PCs to run your operator interface. There are tons of graphics, reports, trends, etc and you use software that is designed to run on Windows, which most of your operator interfaces are designed to do.

When a patch like this hits, the operator interface or historian has issues, but the PLC running the process keeps doing it's job, you just can't see into the PLC.

So yes and no. There are things that are more complex and that could be simplified/run separate from windows, but those start getting prohibitively expensive and the tiny bit of extra reliability is not needed. Those kinds of systems cost 2-5 times as much and the development of those systems is more expensive because there are even fewer people with experience with it. If I had experience with those systems, I would be making 70% more then I am now and I am making enough that I don't need to complain.

Re:Industrial systems should be super-simple

[EndlessNameless]

This has always been an option, and the industry chose Door #1 years ago because it is by far the most productive and economical.

It's not like the machinery is getting any simpler either. How much data do they send and how often do they need to report? Do they integrate with inventory systems to track usage of raw materials and other consumables? How good are the predictive wear/failure alerts?

More precision and more complex automation are going to push those 1970s-era control systems from outdated to unworkable---if it hasn't happened already.

Granted, all industries are not affected the same way. But you're still fundamentally stuck with buying a mass-produced configurable SCADA supervisor for a reasonable price vs a custom system at an insane price. And the mass-produced system is going to have a bit of that "everything and the kitchen sink" flavor.

Re:Industrial systems should be super-simple

[Anonymous Coward]

The PLC can keep the factory lines cranking out widgets into a pile, but you need Wonderware/SCADA for the materials/inventory/shipping/etc. I'm not even sure if a modern factory can execute "we need 1000 X1 widgets followed by 1000 X2 widgets". Maybe if they had all 3 shifts show up for 1 shift to manually setup & trigger the lines with radios to ensure the X1/X2 widget has all the parts it needs in the correct config.

Re:Industrial systems should be super-simple

[Aighearach]

As a software consultant I can inform you, you understood door 1 just fine. But you didn't really understand door 2 very well. It does actually scale, you just fell off your knowledge cliff. When you used the word "specified," that includes scaling.

Re:Industrial systems should be super-simple

[Aighearach]

Door 5 is cheaper, is clearly used for a similar purpose by companies larger than yours, but the manual in only in Chinese and the marketing documents in "English" don't make any sense.

Re:Industrial systems should be super-simple

[Rogue974]

A little off on your description. SCADA is Supervisory Controls and Data Acquisition. There are several parts and pieces to that and in most systems, that included the operator interface which is usually run on Windows machines.

Yes, the equipment that interfaces with the field equipment is fine, but the operators can't see what that equipment is doing.

It would be like saying, your car is fine and the engine is running, but your brakes, gas pedal, steering wheel all stopped responding and your windshield is covered in dirt so you can't see. Engine is fine though!

I work in a facility as a controls engineer that has Wonderware and Rockwell software and I use on a daily basis the software affected by these patches. We didn't path because we don't patch until the vendors vet out patches and say it is ok and we also received the notice that said don't apply the patch.

I know of other facilities that went down because they applied these patches. Yes, the PLCs and controllers were still working, but you can't run blind. Even if you could, the historians have the data you need for EPA compliance or to certify your product for customers so when that goes down, you stop running.

Re:Industrial systems should be super-simple

[Aighearach]

This has always been an option, and the industry chose Door #1 years ago because it is by far the most productive and economical.

Three competing companies are on the same street; one chose door #1, the others chose door #2. One of the ones who chose door #2 has higher productivity and TCO than #1, because the engineering consultant they hired delivered what they promised. The other third company is about to be shut down and liquidated because their consultant never delivered, and after throwing good money after bad, they eventually opened door #1 but because of their debts they didn't have the cash flow to buy large enough quantities of supplies to compete with either of the other two.

In the end, the top companies with the best productivity and the highest profit margin chose door #2. As did many who failed. And most of the production is done by companies who played it safe and went with #1. But even bigger are foreign companies who didn't do any of that, they have a whole different system of doors in their country.

Re:Industrial systems should be super-simple

[whoever57]

Those '70s era microcontrollers easily fit on a cheap FPGA, allowing full customization.

Re:Industrial systems should be super-simple

[Anonymous Coward]

"Where it gets difficult is when you start using PCs to run your operator interface. There are tons of graphics, reports, trends, etc and you use software that is designed to run on Windows, which most of your operator interfaces are designed to do."

It sounds to me that it's the communications between the PC and the PLC that have failed. Something makes me think that protocol is a 'bit banging' protocol and subject to real-time constraints to work (rather than something like TCP over Ethernet). And for all the naysayers, of course a GUI-based management tool is an effective use. Rarely (ideally, never) does the GUI tool need hard-real-time; only human time.

Suppose the ideal fix is a micro-controller or Pi-like system that speaks random latency TCP to the PC and precise timing to the PLC. PC only works in the soft-real-time / human reaction time space.

Re:Industrial systems should be super-simple

[vtcodger]

Probably at least somewhat true that you don't need much of a computer for many industrial control tasks. And even when not true, do the systems have significant sensitive information that needs to be hidden from other users? Is there any point in patching the software against spectre/meltdown if the patches cause trouble and the malware doesn't represent much of a threat to those systems?

OTOH, getting infrastructure OFF the damn internet as quickly as it can be done seems like a very good idea even if doing so will interfere with a lot of startup company visions of girls, fancy cars, mansions and yachts.

Re:Industrial systems should be super-simple

[sjames]

There's a whole high volume world of microcontrollers out there that are more powerful than the chips of the '70s and available dirt cheap. There'd be even more in use but programmers who can write software for them are slightly more expensive.

It's not just the CPU. The microcontrollers tend to have very simple requirements for support on the board. Some can even be trivially plugged into a breadboard and brought up with 5 or so wires.

Re:Industrial systems should be super-simple

[rkordmaa]

For many systems you can make away with dedicated PLC and use something like TwinCAT - turn your windows PC into PLC and have everything in one box with as much oomph and memory as a PLC could ever want. Of course for the particular system it must be acceptable that when windows goes belly up so does the PLC, doesn't happen all that often but it will happen eventually.

Re:Industrial systems should be super-simple

[Actually, I do RTFA]

Or they could be isolated and unpatched. An airgap is the best security, always.

Re:Industrial systems should be super-simple

[sphealey]

The problem, as Target Corporation learned the hard way, is that control systems and their supervisory PCs are often the attack gateway into corporate financial systems. Yes, I'm sure your entity's control systems are never connected to the Internet and are invulnerable - just as Target's HVAC controllers were standalone, not connected to the Internet, and would never be allowed to communicate with the credit card processing system. Problem is they weren't, they were, and they did. If the control system is based on/is managed by a large scale commercial operating system (Windows, Linux, or Mac) and is within 100 km of an Internet connection is needs security patches.

Re:Industrial systems should be super-simple

[thegarbz]

If you have a system that simple and unimportant that you can throw a desktop computer at controlling it then there will be a properly industrial hardened PLC in your price range. No seriously some proper PLCs from the vendor mentioned in the post run at a lower cost than any computer you can throw at it. Just configure and provide it with 24V.

But chances are if you're having problems with the performance of a Wonderware Historian you're problem is that you have 5000 analogue instruments on site and you're committing their values to the database every few seconds.

Re:Industrial systems should be super-simple

[Rogue974]

Agreed. Even air gapped, a system still needs to have security patches applied.

It doesn't matter how great your protocols are to limit access, included in your policy has to be patching. There are many things you have to consider when securing a network with life critical things attached to it. When and how you apply those patches is just as important as any other parts of the policy to secure your network.

Are my systems at risk from Meltdown and Spectre because I have not patched? Yes. Due to other layers in the security, is it likely that those virus's will get to my system? No. Can we wait until everything had been vetted by the vendor so we can apply them without introducing risk into our system? I am not a diviner so can't answer 100%, but we are betting the answer is Yes.

Re:Industrial systems should be super-simple

[Rogue974]

Yes, like you were agreeing to, to run an industrial controls systems you don't need much power, it is just the SCADA (operator interfaces and historian) where you start needing a lot more and it is very difficult to do without getting into the PC and server areas.

As to controls systems being on the internet, yeah, those people are idiots or dealing with stuff that is non proprietary, non life threatening.

At my place, there is not outside logging into the system. They need troubleshooting, I drive in. Other places don't have that luxury and have to make it so you can VPN into the corporate network and then have a path that gets to your controls systems. If done right, can be pretty secure, but where I work we have decided it isn't a risk worth taking.

There are some places where they put their controls systems on their business network or some with no separation between the internet and their SCADA systems. Those people are just asking for trouble.

Re:Industrial systems should be super-simple

[Rogue974]

This is an interesting question. Air gapped is the best solution, but not always acceptable.

Some places need to allow remote access to their controls systems for troubleshooting purposes because they have few experts and it is impractical to fly your controls engineers all over the place.

Even more common is the need to get historical and inventory information up to the business network real time so people can make proper decisions. Security best practices talk little about air gapping because almost everyone wants the data available on the buisness network. Instead, they focus on a multi layers security approach that includes patching, demilatarized zone, Intrusion threat prevention and detection software, etc.

Good companies will have an approach and weight the risk/benefit and the put in the security. When Stuxnet came out, the CEO of my company sat up and then started asking questions. Before that, we did very little and were at significant risk, but no one was looking at controls systems. Stuxnet opened up the world's eyes so security through obscurity became a whole lot more risky for controls systems. I ended up with a couple of others presenting a plan to the CEO and IT Steering committee a security plan and needs and walked out with permissions to spend a lot of money to get up to industry best practices fast! Some places aren't so lucky.

The last 8 years has been a game changer for controls systems security. It used to not be discussed much. Now, it is forcing more controls engineers to learn about network security and more IT people to look at the controls network differently and work with controls to harden connections. We still have a long way to go, but things are improving generally in industry.

Re:Industrial systems should be super-simple

[davidwr]

Even more common is the need to get historical and inventory information up to the business network real time so people can make proper decisions.

This is where a one-way air gap can come in handy:

Have the "secure" side continuously report real-time data 24/7 to a "less secure" device for recording/reporting over a one-way channel. Use VPNs or other controls to give access to the "recording/reporting box" as needed.

If the reporting/reporting box gets compromised, at least it can't directly leapfrog back to the "secure side."

a lot of the manufacturing stuff is stuck in the d

[Joe_Dragon]

a lot of the manufacturing stuff is stuck in the direct hardware access ideas of dos / win 3.1.
Someone needs to do an DirectX / opengl like layer for this stuff to use.

VMware pulled some of their patches

[El Cubano]

VMware pulled some of their patches

Note: ESXi patches associated with VMSA-2018-0004 have been pulled down from the online and offline portal.

...

For ESXi hosts that have not yet applied one of the following patches ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG, VMware recommends not doing so at this time. It is recommended to apply the patches listed in VMSA-2018-0002 instead.

...

For servers using the Intel Haswell and Broadwell processors (see Table 1 for the specific list of affected VMware vSphere supported Intel Haswell and Broadwell processors) that have applied ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG VMware recommends the following:

...

VMware is working closely with Intel and the industry to come to a quick resolution of this Intel microcode issue and provide an update to our customers as soon as possible.

Oops!

reference

Re:VMware pulled some of their patches

[El Cubano]

I guess I should have finished my thought.

It's not just industrial control systems, but hypervisors, and plain old systems too. It sees like this is an object lesson in how speed (in terms of releasing a fix) comes at a cost of performance/quality. I know people were all in a panic once Meltdown and Spectre became public, but this wasn't just fixing a SQL injection vulnerability in Rails or Django. This fundamentally affected the execution of nearly every instruction to go through affected CPUs.

I suspect that the severity and publicity made a more organized roll out with extensive beta testing impossible for just about every vendor that had affected products.

Re:VMware pulled some of their patches

[Anonymous Coward]

On June 1, Intel was informed about Spectre. 3 weeks later about Meltdown. 6-7 months later knowledge became public.

If there is still no organized roll out by then, that can only mean one of two things.

- it is extremely difficult, or even impossible, to fix. Either with our without introducing stability problems. This has huge consequences. Certainly for Intel and maybe for others. It means admitting your products are beyond repair and would result in huge claims which could spell the end of the company,

- Or the sense of urgency at Intel, and maybe its software partners, was lacking.

Re:VMware pulled some of their patches

[Anonymous Coward]

The crazy thing was that Spectre and Meltdown were only disclosed a few days ahead of the industry’s established schedule. These people had /months/ to get patches ready and make sure they weren’t going to f everything up.

Something tells me nobody planned to actually patch anything, and were hoping to get an “extension” at the last minute.

So don’t bother to give these guys the benefit of the doubt. They sat on this for a good long time and are only NOW scrambling and trying to publish patches to fix broken patches to fix broken patches several times a day because they weren’t handling it appropriately since the very beginning.

Stuxnet is now crashing

[Anonymous Coward]

Now things like Stuxnet won't be able to infiltrate as easily. WTF are these things doing connected anyway, and if not connected why do they need the patches? And don't get me started on Windows...

Re:Stuxnet is now crashing

[Anonymous Coward]

Local insider attacker might try to pry a password for something the attacker doesn't have an access. Patching the issue eventually is a good idea. And to our collective amazement, until recently quite a few facilities had their control networks directly attached to the Internet. Even if only the office networks would have Internet access, the control side still wants to have nice things from the office and therefore the connection is there again.

Re:Stuxnet is now crashing

[thegarbz]

WTF are these things doing connected anyway

So to answer your question from multiple points:

Stuxnet : Stuxnet did not infect a connected system. It was spread by direct attack via USB.
Connected : A wonderware historian serves the sole purpose of storing long term trends of your entire control system. This is very useful data ... if you have access to it. This is very sensitive data which some governments will require you to store in realtime ... offsite.

Now to make a point: On a well setup and secure site the Historian will sit on a different network than the control system and receive data through some specific channel, either one way firewall (data diode) or through some other DMZ. The historian itself should sit on a DMZ specifically so that people have no requirement to access the control system.

The real-time live data from the historian is critical information to drive real-time business decisions and often needs to be accessible not only away from the plant, but outside of the building, or even on the other side of the world. Good luck doing that without some "connection".

Ironic

[ilsaloving]

So the meltdown patches are themselves causing meltdowns? Isn't it ironic! (Doncha think?)

Re:Ironic

[Mal-2]

Like rain on your wedding day.

Re: Ironic

[Anonymous Coward]

A freeeeeeee ride when ya alreadyyyyyy paidddddddddddd.

Reliability above all else

[sjbe]

a lot of the manufacturing stuff is stuck in the direct hardware access ideas of dos / win 3.1.

That's not necessarily a bad thing. Manufacturing equipment generally doesn't benefit greatly from having the latest and fanciest do-dads and gee-gaws. It needs to be reliable above all else in most cases. Well made manufacturing equipment mostly should never need to be patched. There are some exceptions but they are rare. You do run into some networked CNC and robotics that needs to be more up to date but the presses that my company runs really just need to go up and down. More complexity or features would not add any value to them.

I run into all sorts of tooling that has lots of nifty features that never get used. I have a wire cutting machine 20 feet from me that does all sorts of stuff to manage wire libraries and is very programmable. None of it gets used. Partly because the interface sucks but mostly because we need something that we can just set up quickly on the fly and start making parts. The device is needlessly complicated and thus more costly than it had to be.

Someone needs to do an DirectX / opengl like layer for this stuff to use.

Sounds like a programmer who doesn't know how manufacturing actually works. If the only tool you have is a hammer...

People use windows for industrial automation???

[exabrial]

People use windows for industrial automation??? Scary world!

Re:People use windows for industrial automation???

[Anonymous Coward]

I did - More for monitoring/kicking off processes - aka "Soft" real time - where if things went wrong, nothing bad would happen
"Hey, Controller at location XXX, please kick off your embedded program #2", and once a minute, I'd ask "Hey, what is the current parameters?" and log them to disk. Saved having to have to go out to chart recorders, etc, and see what was going on from my desk - put it up on the dashboard

Re:People use windows for industrial automation???

[Anonymous Coward]

Usually only for the user interface.

Since Windows isn't a hard Realtime OS, anything that does run on a windows box runs alongside Windows. You can even hide an entire core and memory block from windows.

Cue the inevitable "BSOD" example where windows crashes and restarts, and all the while the PLC logic is still running.
Not scary at all.

Re:People use windows for industrial automation???

[Rogue974]

Yes, absolutely, most industrial automation are run on Windows.

No, it isn't that scary. It can be, but if properly implemented with the right security in mind, you can keep the system up and running reliably.

As stated below, the windows machines are used for the operator interfaces and to record information. The things that actually controls the process are different and unaffected by this and the screwy things with windows.

I am a controls engineer, i.e. program, spec, maintain, industrial controls systems for a living. I work with 4 others and we have a combine over 120 years experience doing this and between the 5 of us have seen hundred of manufacturing facilities. Yes, Windows do occasionally make us want to throw PCs out the window, but properly implemented the Windows box going down rarely is a big cause for concern as long as you can get it up quickly.

With that said, where I work we have been pushing to go over to servers and thin client implementations, but still running Windows servers as opposed to Win CE, 95, 98, XP, Vista, 7, 10 (we skipped 8)....yes, I have set up installed and troubleshoot industrial software on all of those.

Re:People use windows for industrial automation???

[rkordmaa]

Linux has pretty much zero footprint in industrial automation, if you have a full OS running somewhere its going to be windows.

Re:People use windows for industrial automation???

[rkordmaa]

BSOD is not that much scarier than your regular power outage, with much the same results really.

Lenovo T440s

[Anonymous Coward]

Meltdown patch has caused about 20 of our Lenovo T440s notebooks to BSOD on Win10. Had to uninstall, so were all vulnerable and awaiting Lenovo...

Re: Lenovo T440s

[Anonymous Coward]

YEah Win 3.2 running legacy hardware inside manufacturing plant that still use serial and ISA busses.

Toldja so...

[GerryGilmore]

From the very beginning, I've tried to get everyone to pause the Panic Parade, but nnnnnooooooo. To try to address probably the most complex vulnerability yet discovered (it took over 20 YEARS for this to be found) that also requires you to already be running malware on your system, people are flashing new BIOSes, patching kernels and generally behaving like idiots. Slow FT down, folks! Let the CPU and OS experts have a real shot at minimizing the risk, without killing our production systems, FFS!!

Re:Toldja so...

[Anonymous Coward]

rhel and centos updates just work...

Re:Toldja so...

[Anonymous Coward]

From the very beginning, I've tried to get everyone to pause the Panic Parade, but nnnnnooooooo. To try to address probably the most complex vulnerability yet discovered (it took over 20 YEARS for this to be found) that also requires you to already be running malware on your system, people are flashing new BIOSes, patching kernels and generally behaving like idiots. Slow FT down, folks! Let the CPU and OS experts have a real shot at minimizing the risk, without killing our production systems, FFS!!

Just because it took 20 years to be discovered doesn't mean that it is going to take long for hacking toolkits to include exploits based on the PROOF OF CONCEPT CODE RELEASED BY THE DISCOVERERS. There are three variations of the Spectre exploit which allow access to memory which shouldn't be accessible outside the attacking code, the most worrying one is Meltdown which affects all Intel CPUs that use Out of Order Execution which allows access to protect kernel memory. It can be exploited via something as simple as Javascript. This is the variation that these patches are designed to mitigate. The other two variations which affect most CPUs are less worrying as they don't allow access to protected memory but are also harder to mitigate against as each individual program requires patching to protect against attacks. Not all programs require protection against Spectre attacks, your calculator or solitaire app does not have access to any information that you would care about losing.

If I were someone who liked to hack computers for lulz and profits, I would be scrambling to root as many systems via the Meltdown exploit as I could while the shitshow of patching carries on...

Re:Toldja so...

[Anonymous Coward]

Why haven't the people who hack computers for lulz and profits doing that?

Because it is just as complicated to exploit this issue as it was to detect and fix it.

And those people (script kiddies) don't have the talent to do so.

Show me your javascript exploit, please, instead of talking out of your ass.

Re:Toldja so...

[Anonymous Coward]

To try to address probably the most complex vulnerability yet discovered (it took over 20 YEARS for this to be found) that also requires you to already be running malware on your system

That's awfully mean, here we just call them "end users"

I'm happy for you that all of your computers only have one user account on them which is yourself, and I mean that without sarcasm, but unfortunately not all of us are as lucky with all of our systems.

Re:Toldja so...

[edtice1559]

I guess this has somewhat of a political slant, but I don't think a -1 moderation is really fair. The reality is that much of society transitioned from healthy skepticism to assuming that experts are always wrong. That's unfortunate because we need experts *especially* in situations where the experts are most likely to be wrong. Even when experts are "wrong" they usually provide advice that is reasonable enough that we can recover when new information emerges. But the fact is that being unqualified is suddenly a qualification.

Re:Toldja so...

[duke_cheetah2003]

Yeah, pretty much everything GerryGilmore said. This knee-jerk reaction to a pretty obscure flaw is way overboard.

I personally don't want my CPU's branch prediction gimped because some other idiot can't keep his web browser away from malicious sites.

The only panic that should be realistic and warranted is big cloud VM providers concerned these attacks could compromise tenants on shared systems. Patch that shit into oblivion as far as I'm concerned, but get your grubby patches off my desktop. I don't want it, I'll take the performance any time of the day, I know how to avoid questionable web sites and content.

Re:Toldja so...

[Anonymous Coward]

This wasn’t just found back in the summer. There have been many academic papers saying “don’t do speculative execution, because security” published and presented in the past. The only thing that changed over the summer is that someone finally wrote a POC and demonstrated that this “theoretical vulnerability” was a real one.

Re:Toldja so...

[Man On Pink Corner]

Just because it took 20 years to be discovered doesn't mean that it is going to take long for hacking toolkits to include exploits based on the PROOF OF CONCEPT CODE RELEASED BY THE DISCOVERERS.

Still waiting for a PoC that isn't just a rigged video that someone says is a PoC.

Show me a rogue Web page that reads passwords or other private data, and I'll panic.

This will linger for decades

[Anonymous Coward]

Having some experience around these kinds of systems they donâ(TM)t change once they are productionized until end of life which can be 2 decades or more. Donâ(TM)t put control systems and manufacturing machines on an internet connected network ever. Thatâ(TM)s the beginning middle and end on this topic.

Re:This will linger for decades

[gettin2old]

Agreed. I've worked with a lot of these systems. And most companies connect them to their internal network because all the PLCs are network ready. The funny thing is most of the controls PCs don't talk to any corporate resources except possibly for authentication. (even then the users typically all log in to only one controller account). I'd bet in 90% of the cases an isolated network switch and a few patch cables solves their problems.

Also Beckhoff TwinCAT 3

[RobinH]

We received a notification from Beckhoff to avoid these patches for TwinCAT 3 until they would patch their runtime to be compatible. We update through WSUS so we were able to do that. Beckhoff themselves urge you *not* to install Windows Updates on their control system PCs even though they bill their product as part of the "Internet of Things" and play up the connectivity of everything. They're hypocrites, but Rockwell did the same thing when we used their product. They wouldn't warranty their software if you installed anti-virus on the same server as their historian product.

Re: Also Beckhoff TwinCAT 3

[Zero__Kelvin]

It is not hypocritical for them to warn against the dangers of applying Windows updates, especially for industrial applications. Microsoft has a well established pattern of changing functionality without warning via patches and they Bork systems on a regular basis.

Re: Also Beckhoff TwinCAT 3

[RobinH]

It's absolutely hypocritical of them to sell a Windows PC-based industrial control system, and instead of maintaining their product and testing it with each new Windows Update, they just put out a blanket statement to turn off Windows Updates and put it behind a firewall, and then sell their product based on the connectivity it provides. I'm OK with delayed patch installation and extra security measures, but every patch needs to be tested by them and certified for installation. They have no mechanism for doing that at all.

Re: Also Beckhoff TwinCAT 3

[rkordmaa]

Most industrial systems are built once and then left well alone to not muck anything up, periodic Windows updates and accompanying mandatory restarts are pretty much a non starter in most cases. If it works, don't touch it, is the wisdom when it comes to industrial systems.

Re: Also Beckhoff TwinCAT 3

[Rogue974]

Quote, "I'm OK with delayed patch installation and extra security measures, but every patch needs to be tested by them and certified for installation. They have no mechanism for doing that at all."

They actually do have a mechanism for testing and releasing what patches are acceptable on their systems. This articles talks about Wonderware and Rockwell systems, both of which I use on a daily basis as an end user. Both make available a list of what patches they have tested and vetted out for their systems. It is a pain in the butt to shift through their databases on their websites and you have to have support contracts with them to get to the information, but both companies do exactly what you say they do not do.

They do recommend turning off windows update and keeping it away from the internet and your business network as part of the security model, but that is a gross simplification of industry best practices and what they recommend.

Re: Also Beckhoff TwinCAT 3

[RobinH]

That is the old model before all the data collection requirements arrived. The new model needs data connectivity, so patches are a must, or else you need expensive, complicated and difficult-to-implement things like data diodes.

Industrial systems don't have as much spare room

[plague911]

I have never worked on industrial systems, I did work on some large scale defense equipment. One of the design considerations is cost, in order to minimize cost, you match the components spec to the semi-well defined performance need. No need on buying a V12 when a V6 will do...... Now I am not saying you don't build in some buffer, but the MASSIVE performance hit required by these patches could easily blow the given performance buffer out of the water. I could easily see how billions of dollars worth of industrial systems simply will not be able to patched due performance cost of the patches. Additionally given the age/design of the systems there is no way to conveniently upgrade the systems.

up to date to just run on newer boards is one thin

[Joe_Dragon]

up to date to just run on newer boards is one thing. The last boards to do ISA out side of SBC systems are OLD p4's.

Maayyybe

[DeplorableCodeMonkey]

If your control systems weren't connected to the Interwebz so you could save a little money on payroll, you wouldn't have to apply the patch.

Re:Maayyybe

[Rogue974]

Yeah, not so much.

Controls systems not connected to the internet still need to be patched and maintained because there are vectors of attack that can still get across an air gap.

Yes, patching isn't as important, but you still have to patch for security and just to be able to stay compliant with software revisions of the software you are using.

FYI, I am a controls engineer, that means I do this for a living. I use the software mentioned that this crashes, but it didn't hit me because I would never apply a patch until it had been tested and approved by the vendor. This patch was not. As a matter of fact, they sent a notice to all of their customers to tell them not to apply this patch because it takes their software down!

Re:Maayyybe

[Bob the Super Hamste]

Says the imbecile who is not aware of how security works. Just ask the Iranians about how not connecting ICS systems to the internet worked. Yes there are plenty of dumb managers out there who would want to connect these systems to the corporate network (and thus the internet) to save a buck but then there are methods of attack that can work around even an air gapped one. The people who are actively targeting these systems are not just your run of the mill script kiddies, or computer crime orgs, but often are nation states. That is why orgs like NERC have patching requirements for systems even though those systems aren't suppose to be connected to the internet. Layer upon layer of security, so you know if Bill the janitor finds a USB stick conveniently left on the floor near by and then tries to plug it into a server it won't cause issues, or Bill instead is just a bad actor.

Deploy

[M0j0_j0j0]

User:*Computer install patch and do regression test.

Computer: The company has now regressed

The hazards of monoculture

[plopez]

relying on one piece of tech is as bad as relying on one food crop.

Re:The hazards of monoculture

[Anonymous Coward]

relying on one piece of tech is as bad as relying on one food crop.

this is why humans are gonna go extinct, they are all the same

Re:As a deeply conservative Christian I am appaled

[plopez]

If you're not living by the example of Jesus (The man from the middle east, not the guy who trims the hedges) you're not a Christian.

Re:a lot of the manufacturing stuff is stuck in th

[RobinH]

That's what OPC UA is, from my understanding. I never like original OPC due to the performance hit, but I understand that OPC UA is much better.

Re:you are a pathetic idiot

[Anonymous Coward]

> yeah cuz turning relays on and off requires so much cpu!

CPU time versus latency. If all syscalls end up becoming slower, then that's a latency problem on every such event. For example if industrial control systems need a lot of IPC, then that's a lot of syscalls.

Latency versus bandwidth is often explained as a truck filled up with tapes has a huge amount of bandwidth, but a very bad latency. This here is similar: a very fast CPU has a huge amount of bandwidth, the Meltdown patches cause bad latency.

Re:Simple solutiton

[Anonymous Coward]

But then they'll get framerate drops in World of Warcraft.

Re: Industrial systems don't have as much spare ro

[Zero__Kelvin]

Your whole diatribe is ridiculous. The parts of the system to which you refer aren't even the ones affected, nor would they be the way you describe it. These are the GUI based interfaces that are experiencing the issue, not the core system itself as you falsely assumed.

you are an even worse pathetic idiot

[Anonymous Coward]

the Meltdown patches cause bad latency.

no, they do not. the latency is not measurable compared to the latency of mechanical relays and optoisolators

but feel free to continue to show what an ignorant idiot you are

No they shouldn't, simplicity is too complicated

[Anonymous Coward]

an 8-bit micro-controller won't cut it anymore.

Everything is so precision motion-based and customizable.
Everything needs to have comprehensive diagnostics to say what's wrong (because troubleshooting has been replaced with magical thinking).
Everything is running at multiples of 125 microseconds, now even update rates of 10 milliseconds is too long.

Serial (RS232) is an absolute nightmare to diagnose. RS485 is a nightmare to get them to wire it correctly. YES, RS485 as in 2 wires and a shield is still an absolute troubleshooting nightmare. Colorblind landing wires wrong, cutting off the shielding, not piercing insulation, exceeding bend radius, signal reflections, bad splices, crushed or overflexed cable... Industrial Ethernet wins hands down for its ability for diagnostic troubleshooting, because what's electrically simple is horrendously complicated when bureaucracies and ignoramus is involved.

Simple isn't simple. You can stuff a 32-bit CPU with megs of RAM into the same simple looking box as a vintage 8051 board. One that nobody is going to bother to desolder, let alone even take apart and diagnose at the component level when it fails.

Re: Industrial systems don't have as much spare ro

[plague911]

"as you falsely assumed" No I did not assume that. Re-read k-thanks bye.

Re:No they shouldn't, simplicity is too complicate

[aaarrrgggh]

Spot-on. Not sure if you are being sarcastic on a couple of the points, but when you have access to second and third or fourth derivative data rather than a simple counter you have much more valuable diagnostic data to work with to streamline a process. It might not be needed continuously, but to have it at all means you get it continuously.

Yes, it can make people lazy-- we will just tune it in the field-- but for some things a 1% delta is really important.

control systems need Linux

[humankind]

If you're running an important control system using Windows, you're doing it wrong.

Re:control systems need Linux

[rkordmaa]

Says the guy who has never created industrial anything. Not that it wouldn't be nice to use Linux, but reality is that Linux might as well not exist on industrial scene, all the software for pretty much anything is Windows only, so good luck working with your Linux based control system.

Re:control systems need Linux

[Anonymous Coward]

Says the guy who has never created industrial anything. Not that it wouldn't be nice to use Linux, but reality is that Linux might as well not exist on industrial scene, all the software for pretty much anything is Windows only, so good luck working with your Linux based control system.

Well, good luck working with your Windows based control system then, you knob.

I've worked on industrial systems

[Anonymous Coward]

Like anything from a distance it looks pretty simple. Get up close and you have to deal with timing problems, safety issues, requirements, timing issues, interfaces to external systems. When they go wrong the consequences are severe than making an coffee while you reboot your PC. If they truly were simple people with 6 months training would be able to do it. As it is in SCADA experience is greatly valued.

Captain flies away

All the more reason

[RightwingNutjob]

not to patch shit just because some code money says to. Security and functionality are in direct opposition to each other. Increasing security lowers availability and increases the number of points of failure. This is a trade that real engineers (as opposed to software weenies) make in the presence of limited computational resources. Things that face the public and run unattended in remote installations? Err on the side of security. Things that never see the light of day and run inside a steel vault with two guards stationed outside at all times? Fuck IT security.

Re:No they shouldn't, simplicity is too complicate

[sjames]

So go with a 32 bit microcontroller with an ethernet port.

Wut?

[Anonymous Coward]

Mission critical industrial control systems use Microsoft updates? Tough sh!t then. I guess they didn't read the Microsoft EULAs and disclaimers.

Now you tell me

[Chrisq]

I've just installed it in the nuclear reactor control system. The IT security department inist.......auuugh!

Re:Industrial systems don't have as much spare roo

[thegarbz]

I could easily see how billions of dollars worth of industrial systems simply will not be able to patched due performance cost of the patches.

Doubt it. The systems themselves are unaffected by this mess and don't need patching in any case (controlled access and all). What is being patched here is a handful of database servers. Spend the $3000 to buy a better one. In the grand scheme of "industrial control" not only will it not break the bank but you can most likely pay for it using petty cash.