How Are Sysadmins Handling Spectre/Meltdown Patches? (hpe.com)

Esther Schindler (Slashdot reader #16,185) writes that the Spectre and Meltdown vulnerabilities have become "a serious distraction" for sysadmins trying to apply patches and keep up with new fixes, sharing an HPE article described as "what other sysadmins have done so far, as well as their current plans and long-term strategy, not to mention how to communicate progress to management." Everyone has applied patches. But that sounds ever so simple. Ron, an IT admin, summarizes the situation succinctly: "More like applied, applied another, removed, I think re-applied, I give up, and have no clue where I am anymore." That is, sysadmins are ready to apply patches -- when a patch exists. "I applied the patches for Meltdown but I am still waiting for Spectre patches from manufacturers," explains an IT pro named Nick... Vendors have released, pulled back, re-released, and re-pulled back patches, explains Chase, a network administrator. "Everyone is so concerned by this that they rushed code out without testing it enough, leading to what I've heard referred to as 'speculative reboots'..."

The confusion -- and rumored performance hits -- are causing some sysadmins to adopt a "watch carefully" and "wait and see" approach... "The problem is that the patches don't come at no cost in terms of performance. In fact, some patches have warnings about the potential side effects," says Sandra, who recently retired from 30 years of sysadmin work. "Projections of how badly performance will be affected range from 'You won't notice it' to 'significantly impacted.'" Plus, IT staff have to look into whether the patches themselves could break something. They're looking for vulnerabilities and running tests to evaluate how patched systems might break down or be open to other problems.
The article concludes that "everyone knows that Spectre and Meltdown patches are just Band-Aids," with some now looking at buying new servers. One university systems engineer says "I would be curious to see what the new performance figures for Intel vs. AMD (vs. ARM?) turn out to be."

Comments

That's how

[Artem S. Tashkinov]

Both vulnerabilities are blown out of proportions and you need to rush to actively fix them only when your platform runs untrusted code which is mostly relevant for VPS/clouds/etc.

When you only run your own trusted code (say a DB or an HTTP server), there's little if any need to patch them urgently. Of course, this implies that your authentication process is properly secured and when it's not, the intruder might as well find other local unpatched vulnerabilities.

Re:That's how

[tlhIngan]

Trust isn't binary. No code is fully trusted. There is a whole spectrum from core kernel security functions in an open source OS to random Javascript served up by ads.

Most people run some proprietary software. Most people have not carefully security audited all their open source software. That's why operating systems have feature to isolate tasks, to protect the kernel and manage hardware access rights.

For most people the Meltdown patch is essential. Exploits are already in the wild.

No, it's overhyped. Perhaps if you're running a VM and intermix publicly accessible services with internal services, then you will want to worry about meltdown and spectre potentially causing the public VM to grab data from the secure VM. Of course, the other solution can use is to separate the machines physically, so someone exploiting meltdown on your public VM gets access to the other public VMs.

Here, the threat is not from the software on the VM, but from someone finding an exploit in the software and exploiting it. But there is nothing you can run that will get you access to the other private servers, especially with proper firewalling in place.

For single-server machines, the patches aren't as useful - if you break into the server via an exploit and then get root, just because you have patched it against meltdown means nothing - since you can access kernel memory anyways much more easily.

Plus, there are plenty of user-mode meltdown patches out there - the whole javascript exploit is now useless because all the major browsers have made it so "high resolution timers" aren't so high-resolution - they're around the 1msec range which is enough for scripts, but too coarse to actually do a meltdown exploit (the timing difference between cached and uncached is small and 1msec is not fine enough to tell).

The goal is to recognize that the problem is localized to one machine, and it inadvertently allows processes to read memory they're not supposed to. For a VM server, this is bad, since it means once VM can read memory of another VM. For a cloud service provider, this is disastrous since it means an evil VM can read other customer's data.

Within a company, it's a lot less serious if you already have the proper network segregation in place, you don't mix internal and externally accessible VMs on the same machine and other precautions. In a non-VM situation, it's a non-event - exploiting the service grants you access to the machine. And once that's happens, it can be assumed you can access the entire filesystem and everything accessible to the machine anyways.

How deep do most people go down the patch hole?

[swb]

I guess what I'm referring to is digging into every single patch to try to figure out what the fuck it actually patches. And if you *do* get some kind of detail on what a specific patch actually fixes, is the information meaningful enough to decide whether you *should* apply this specific patch (relevance, risk, etc)?

Is it easier or harder now with so many vendors releasing "rollup" patches which contain multiple patches, some of which are all-inclusive and some of which require some previous rollup installed? Now picking and choosing specific patches is more or less out the door.

And then there's the question of whether the vendor even makes it easy/hard to have any control over patches, automatically just giving you patch(es) in some form or other. And of course let's not forget support -- will the vendor provide any support if you are missing patches or do you have to have them all installed anyway?

I guess what I see this boiling down to is "Who cares?" Install all the latest available patches and hope for the best. Only a full-time dedicated patch admin for a narrow product silo has the time/energy/understanding to break down the compound patching environment into something coherent and also probably is also the only one to have a complex patch management system that gives them granular control over which patches get installed and which don't.

Also, based on the last few years of software quality we're all beta testers anyway. Pretty much everything released is beta quality and hits true stability and reliability just about the point the new version is released and taming its worst initial bugs.