How Are Sysadmins Handling Spectre/Meltdown Patches? (hpe.com)

Esther Schindler (Slashdot reader #16,185) writes that the Spectre and Meltdown vulnerabilities have become "a serious distraction" for sysadmins trying to apply patches and keep up with new fixes, sharing an HPE article described as "what other sysadmins have done so far, as well as their current plans and long-term strategy, not to mention how to communicate progress to management." Everyone has applied patches. But that sounds ever so simple. Ron, an IT admin, summarizes the situation succinctly: "More like applied, applied another, removed, I think re-applied, I give up, and have no clue where I am anymore." That is, sysadmins are ready to apply patches -- when a patch exists. "I applied the patches for Meltdown but I am still waiting for Spectre patches from manufacturers," explains an IT pro named Nick... Vendors have released, pulled back, re-released, and re-pulled back patches, explains Chase, a network administrator. "Everyone is so concerned by this that they rushed code out without testing it enough, leading to what I've heard referred to as 'speculative reboots'..."

The confusion -- and rumored performance hits -- are causing some sysadmins to adopt a "watch carefully" and "wait and see" approach... "The problem is that the patches don't come at no cost in terms of performance. In fact, some patches have warnings about the potential side effects," says Sandra, who recently retired from 30 years of sysadmin work. "Projections of how badly performance will be affected range from 'You won't notice it' to 'significantly impacted.'" Plus, IT staff have to look into whether the patches themselves could break something. They're looking for vulnerabilities and running tests to evaluate how patched systems might break down or be open to other problems.
The article concludes that "everyone knows that Spectre and Meltdown patches are just Band-Aids," with some now looking at buying new servers. One university systems engineer says "I would be curious to see what the new performance figures for Intel vs. AMD (vs. ARM?) turn out to be."

Comments

That's how

[Artem S. Tashkinov]

Both vulnerabilities are blown out of proportions and you need to rush to actively fix them only when your platform runs untrusted code which is mostly relevant for VPS/clouds/etc.

When you only run your own trusted code (say a DB or an HTTP server), there's little if any need to patch them urgently. Of course, this implies that your authentication process is properly secured and when it's not, the intruder might as well find other local unpatched vulnerabilities.

Re:That's how

[AmiMoJo]

Trust isn't binary. No code is fully trusted. There is a whole spectrum from core kernel security functions in an open source OS to random Javascript served up by ads.

Most people run some proprietary software. Most people have not carefully security audited all their open source software. That's why operating systems have feature to isolate tasks, to protect the kernel and manage hardware access rights.

For most people the Meltdown patch is essential. Exploits are already in the wild.

Re:That's how

[Z00L00K]

And it doesn't help that some fixes out there has created problems, being recalled and then replaced with a modified version.

As a sysadmin where uptime is important and the servers aren't in a high exposure position it's better to wait and see that things are stable before patching the systems in a panic.

In addition to this - what about the network equipment like routers and switches - aren't they vulnerable as well? Maybe not to the same extent, but some are.

Re:That's how

[Anonymous Coward]

As far as I know no one has still been able to provide a working spectre vulnerability that doesn't rely on custom written victim code,
or specially instrumented code. If I'm wrong, please post me a link to the code.
Meltdown is different. Basically never use intel again until they in 5 years they have fixed it in hardware.
As for running trusted code on anything other than bare metal:
Why would you trust the virtualization layer any more than your trusted code?
That's retarded and a failure in your trust model.
Trust becomes less as you go down the layers further from the hardware.
Whatever runs directly on hardware should be the most trusted code.

Re:That's how

[tlhIngan]

Trust isn't binary. No code is fully trusted. There is a whole spectrum from core kernel security functions in an open source OS to random Javascript served up by ads.

Most people run some proprietary software. Most people have not carefully security audited all their open source software. That's why operating systems have feature to isolate tasks, to protect the kernel and manage hardware access rights.

For most people the Meltdown patch is essential. Exploits are already in the wild.

No, it's overhyped. Perhaps if you're running a VM and intermix publicly accessible services with internal services, then you will want to worry about meltdown and spectre potentially causing the public VM to grab data from the secure VM. Of course, the other solution can use is to separate the machines physically, so someone exploiting meltdown on your public VM gets access to the other public VMs.

Here, the threat is not from the software on the VM, but from someone finding an exploit in the software and exploiting it. But there is nothing you can run that will get you access to the other private servers, especially with proper firewalling in place.

For single-server machines, the patches aren't as useful - if you break into the server via an exploit and then get root, just because you have patched it against meltdown means nothing - since you can access kernel memory anyways much more easily.

Plus, there are plenty of user-mode meltdown patches out there - the whole javascript exploit is now useless because all the major browsers have made it so "high resolution timers" aren't so high-resolution - they're around the 1msec range which is enough for scripts, but too coarse to actually do a meltdown exploit (the timing difference between cached and uncached is small and 1msec is not fine enough to tell).

The goal is to recognize that the problem is localized to one machine, and it inadvertently allows processes to read memory they're not supposed to. For a VM server, this is bad, since it means once VM can read memory of another VM. For a cloud service provider, this is disastrous since it means an evil VM can read other customer's data.

Within a company, it's a lot less serious if you already have the proper network segregation in place, you don't mix internal and externally accessible VMs on the same machine and other precautions. In a non-VM situation, it's a non-event - exploiting the service grants you access to the machine. And once that's happens, it can be assumed you can access the entire filesystem and everything accessible to the machine anyways.

Re:That's how

[drinkypoo]

When you only run your own trusted code (say a DB or an HTTP server), there's little if any need to patch them urgently.

Any server which is remotely accessible, which is all of them, could potentially be vulnerable to some kind of remote code injection flaw in one of its public-facing services. So no. Absolutely no.

Re:That's how

[thegarbz]

You're talking about trust in code. The GP is talking about trust in logged in users.

There's no patch for the vulnerability you describe.

Re:That's how

[Artem S. Tashkinov]

despite there being Javascript exploits in the wild

All major web browsers have already been patched against this vulnerability. Nice joke, though.

They still have sysadmins?

[DNS-and-BIND]

I thought the sysadmin had pretty much been eliminated in favor of outsourcing IT and making the developers do it themselves. It's a prime area for cost cutting, good sysadmins aren't cheap and you won't notice they're gone because they tend to automate their jobs.

Re:Are sysadmins caring about user experience now?

[Mister Liberty]

What caused them to run the virus scanner in the first place?

Re:Hosts File

[Anonymous Coward]

Er, hosts file is incredibly out dated, hell most things ignore the hosts file.

Besides this the hosts file is missing certain tools to make it viable for most things.

It needs wildcards and regex patterns to be worth a damn in todays world where you want to not just block a specific IP but rather a block of IP. Iptables generally does a much better job of this than rink a dinking around with a billion IP addresses which change like the wind blows.

I think you are the guy I see all the time going on silly rants about using a hosts file, it just isn't true, and anyone who really wants to perform blocking is better of using other methods. To anyone who ever reads this APK fellow again, hosts file is NOT the way to go, you should be looking for broader more modern tools that can actually get the job done. This man is just an insane lunatic peddling his own brand of snake oil and it will not help or protect you.

Re:Are sysadmins caring about user experience now?

[drinkypoo]

Go do your f'ing job and install the patches from hell, I say. And if the drop in performance bothers you, maybe we can finally talk about turning down the virus scanner to a normal level of security.

You do realize that it's the users that are always clamoring for more power? Sysadmins were happy with the 68k.

How deep do most people go down the patch hole?

[swb]

I guess what I'm referring to is digging into every single patch to try to figure out what the fuck it actually patches. And if you *do* get some kind of detail on what a specific patch actually fixes, is the information meaningful enough to decide whether you *should* apply this specific patch (relevance, risk, etc)?

Is it easier or harder now with so many vendors releasing "rollup" patches which contain multiple patches, some of which are all-inclusive and some of which require some previous rollup installed? Now picking and choosing specific patches is more or less out the door.

And then there's the question of whether the vendor even makes it easy/hard to have any control over patches, automatically just giving you patch(es) in some form or other. And of course let's not forget support -- will the vendor provide any support if you are missing patches or do you have to have them all installed anyway?

I guess what I see this boiling down to is "Who cares?" Install all the latest available patches and hope for the best. Only a full-time dedicated patch admin for a narrow product silo has the time/energy/understanding to break down the compound patching environment into something coherent and also probably is also the only one to have a complex patch management system that gives them granular control over which patches get installed and which don't.

Also, based on the last few years of software quality we're all beta testers anyway. Pretty much everything released is beta quality and hits true stability and reliability just about the point the new version is released and taming its worst initial bugs.

Re:We do it the easy way ...

[fisted]

True that. I mean, what's more secure than a system that can't boot /and/ resists troubleshooting?

"But fisted", you say, "Look into journalctl." [...] "Oh, you're running cyrus and the mail subsystem produces awful amounts of logs so journalctl will take minutes to even get you to the pager?" "I guess you could.. no, not that, but, maybe grep on journalctl -f?" "What do you mean, you need to see past events?" "Well don't run such a stupid mail server, it's not systemd's fault!!!" "What do you mean, it was not a problem pre systemd?" "LA LA LA I CAN'T HEAR YOU"

Re: We do it the easy way ...

[Zero__Kelvin]

You use /var/log/syslog just like with non-systemd systems. DOH!

Re:Are sysadmins caring about user experience now?

[johannesg]

ISO-9000 compliance, I believe. Having "appropriate security measures" in place is mandated.

Re:Hosts File

[postbigbang]

Hmmm. Lemme see, uses a hosts file.

Ok, we'll poison arp cache. Easy enough.

If that doesn't work, we'll poison DNS.

Or, we'll code inject.

Or we'll spoof theupdates address, or see if the syslogd is correctly configured.

Hmmm, that didn't work? Let's see what ports are open. I wonder if it'll swallow out of sequence mis-formed barrages of packets.

That didn't work? We'll get a guy to install a wallwartPC with an address that's the broadcast for that segment of the VLAN.

Where there's a will, there's a way. Yes, there were lots of hideous CVEs out there before either Intel processor fault. But this one's a doozy. Ultimately, it means you're going to have to buy more hardware, it's just a matter of time. See, servers and PCs were on the decline, and well, there was some revenue that needed peaking back up again. So have a nice day, and just open up that purchase order app and quit bitchin.

Re:Starting up their old Sparcs OFC!

[mdhoover]

Sparc is affected by spectre unfortunately.

Re:Are sysadmins caring about user experience now?

[KozmoStevnNaut]

As much as I love the connectedness of the internet, I also kind of miss the old days of DOS gaming, Win3.11, the early days of Linux and hardware without built-in backdoors ala Intel ME.

Maybe it's just nostalgia and rose-tinted glasses, but life was a lot less complicated then.

Ron's experience sounds a lot like mine.

[sabbede]

Panic, patch, patch, panic, remove patches, reappy patches, panic, remove patches, deal with screaming users, patch, curse Intel....

Re:Are sysadmins caring about user experience now?

[drinkypoo]

Maybe it's just nostalgia and rose-tinted glasses, but life was a lot less complicated then.

The complications were different. You had to fiddle with hypermodem or xzmodem or UUCP, you had to know the AT command set, you had to know arcane technical details of the PC ISA just to get a sound card working.

Ryzen

[tigersha]

Our company needed a new web server in any case, so I figured the time was to take Ryzen for a spin. Very happy with it!

Multi tiered rollout

[Archfeld]

As some have undoubtedly pointed out the true vulnerability of your system depends on exposure and the type of code run on the system, but the idea of patching only a certain segment is less than appetizing. Any business of any size has a tier of test/QA , maybe one of dev, and finally a production line. Obviously you patch one set, test/QA or dev. allow your developers to abuse the hell out of it hopefully, then roll it out to production. The company I currently work for actually has QA engineers who follow a developer provided script and pounds away at any modified branch of code ensuring all the functions do in fact function and they compare results to ensure standard replies etc. It is quite refreshing to see it done correctly for a change.