How Are Sysadmins Handling Spectre/Meltdown Patches? (hpe.com)

Esther Schindler (Slashdot reader #16,185) writes that the Spectre and Meltdown vulnerabilities have become "a serious distraction" for sysadmins trying to apply patches and keep up with new fixes, sharing an HPE article described as "what other sysadmins have done so far, as well as their current plans and long-term strategy, not to mention how to communicate progress to management." Everyone has applied patches. But that sounds ever so simple. Ron, an IT admin, summarizes the situation succinctly: "More like applied, applied another, removed, I think re-applied, I give up, and have no clue where I am anymore." That is, sysadmins are ready to apply patches -- when a patch exists. "I applied the patches for Meltdown but I am still waiting for Spectre patches from manufacturers," explains an IT pro named Nick... Vendors have released, pulled back, re-released, and re-pulled back patches, explains Chase, a network administrator. "Everyone is so concerned by this that they rushed code out without testing it enough, leading to what I've heard referred to as 'speculative reboots'..."

The confusion -- and rumored performance hits -- are causing some sysadmins to adopt a "watch carefully" and "wait and see" approach... "The problem is that the patches don't come at no cost in terms of performance. In fact, some patches have warnings about the potential side effects," says Sandra, who recently retired from 30 years of sysadmin work. "Projections of how badly performance will be affected range from 'You won't notice it' to 'significantly impacted.'" Plus, IT staff have to look into whether the patches themselves could break something. They're looking for vulnerabilities and running tests to evaluate how patched systems might break down or be open to other problems.
The article concludes that "everyone knows that Spectre and Meltdown patches are just Band-Aids," with some now looking at buying new servers. One university systems engineer says "I would be curious to see what the new performance figures for Intel vs. AMD (vs. ARM?) turn out to be."

Comments

That's how

[Artem S. Tashkinov]

Both vulnerabilities are blown out of proportions and you need to rush to actively fix them only when your platform runs untrusted code which is mostly relevant for VPS/clouds/etc.

When you only run your own trusted code (say a DB or an HTTP server), there's little if any need to patch them urgently. Of course, this implies that your authentication process is properly secured and when it's not, the intruder might as well find other local unpatched vulnerabilities.

Re:That's how

[AmiMoJo]

Trust isn't binary. No code is fully trusted. There is a whole spectrum from core kernel security functions in an open source OS to random Javascript served up by ads.

Most people run some proprietary software. Most people have not carefully security audited all their open source software. That's why operating systems have feature to isolate tasks, to protect the kernel and manage hardware access rights.

For most people the Meltdown patch is essential. Exploits are already in the wild.

Re:That's how

[Anonymous Coward]

If the machine in question is vulnerable - and like you say, that's a big "if" - then the first port of call would be to test on dev machines for stability/side effects and to run extensive load tests on your performance environment to work out the effect on the load.
Once a team can quantify the drop in performance, they can look to buy new hardware if necessary, or if they're lucky they'll have sufficient margin on the SLAs to absorb it. There may be some other work that could help on that front - look to get source systems to supply earlier if it's a batch environment etc.

All in all, it shouldn't be a major concern - just another rock hurled at the IT team.

Re:That's how

[Anonymous Coward]

Almost nobody runs trusted code on bare metal these days. Not even banks. Everyone working in this industry has to worry about "VPS/clouds/etc" security. So does every human who relies on any Internet services whatsoever. Your lack of concern is well out of proportion, and only illustrates your having no clue what you are talking about.

Re:That's how

[Z00L00K]

And it doesn't help that some fixes out there has created problems, being recalled and then replaced with a modified version.

As a sysadmin where uptime is important and the servers aren't in a high exposure position it's better to wait and see that things are stable before patching the systems in a panic.

In addition to this - what about the network equipment like routers and switches - aren't they vulnerable as well? Maybe not to the same extent, but some are.

Re:That's how

[Anonymous Coward]

As far as I know no one has still been able to provide a working spectre vulnerability that doesn't rely on custom written victim code,
or specially instrumented code. If I'm wrong, please post me a link to the code.
Meltdown is different. Basically never use intel again until they in 5 years they have fixed it in hardware.
As for running trusted code on anything other than bare metal:
Why would you trust the virtualization layer any more than your trusted code?
That's retarded and a failure in your trust model.
Trust becomes less as you go down the layers further from the hardware.
Whatever runs directly on hardware should be the most trusted code.

Re:That's how

[tlhIngan]

Trust isn't binary. No code is fully trusted. There is a whole spectrum from core kernel security functions in an open source OS to random Javascript served up by ads.

Most people run some proprietary software. Most people have not carefully security audited all their open source software. That's why operating systems have feature to isolate tasks, to protect the kernel and manage hardware access rights.

For most people the Meltdown patch is essential. Exploits are already in the wild.

No, it's overhyped. Perhaps if you're running a VM and intermix publicly accessible services with internal services, then you will want to worry about meltdown and spectre potentially causing the public VM to grab data from the secure VM. Of course, the other solution can use is to separate the machines physically, so someone exploiting meltdown on your public VM gets access to the other public VMs.

Here, the threat is not from the software on the VM, but from someone finding an exploit in the software and exploiting it. But there is nothing you can run that will get you access to the other private servers, especially with proper firewalling in place.

For single-server machines, the patches aren't as useful - if you break into the server via an exploit and then get root, just because you have patched it against meltdown means nothing - since you can access kernel memory anyways much more easily.

Plus, there are plenty of user-mode meltdown patches out there - the whole javascript exploit is now useless because all the major browsers have made it so "high resolution timers" aren't so high-resolution - they're around the 1msec range which is enough for scripts, but too coarse to actually do a meltdown exploit (the timing difference between cached and uncached is small and 1msec is not fine enough to tell).

The goal is to recognize that the problem is localized to one machine, and it inadvertently allows processes to read memory they're not supposed to. For a VM server, this is bad, since it means once VM can read memory of another VM. For a cloud service provider, this is disastrous since it means an evil VM can read other customer's data.

Within a company, it's a lot less serious if you already have the proper network segregation in place, you don't mix internal and externally accessible VMs on the same machine and other precautions. In a non-VM situation, it's a non-event - exploiting the service grants you access to the machine. And once that's happens, it can be assumed you can access the entire filesystem and everything accessible to the machine anyways.

Re:That's how

[Anonymous Coward]

your platform runs untrusted code which is mostly relevant for VPS/clouds/etc.

Or, say, a Javascript engine in a web-browser?

Re:That's how

[drinkypoo]

When you only run your own trusted code (say a DB or an HTTP server), there's little if any need to patch them urgently.

Any server which is remotely accessible, which is all of them, could potentially be vulnerable to some kind of remote code injection flaw in one of its public-facing services. So no. Absolutely no.

Re:That's how

[thegarbz]

You're talking about trust in code. The GP is talking about trust in logged in users.

There's no patch for the vulnerability you describe.

Re:That's how

[Anonymous Coward]

Russian sounding last name. Check. Suggests regular users don't need to patch Spectre and Meltdown, despite there being Javascript exploits in the wild. Check.

I'm not going to fall for it and let you read all of my kernel memory, Vladamir. :P

Re:That's how

[Artem S. Tashkinov]

despite there being Javascript exploits in the wild

All major web browsers have already been patched against this vulnerability. Nice joke, though.

Are sysadmins caring about user experience now?

[johannesg]

After decades of struggling with virus scanners that insisted on slowly, laboriously scanning every .h file on every access during every compile, the insistence of sysadmins on braindead security policies has already wasted months of my life. I guess my only question is: what's different now? Is it, perhaps, that they themselves would also be bothered by it this time?

Go do your f'ing job and install the patches from hell, I say. And if the drop in performance bothers you, maybe we can finally talk about turning down the virus scanner to a normal level of security.

Re:Are sysadmins caring about user experience now?

[Mister Liberty]

What caused them to run the virus scanner in the first place?

Re:Are sysadmins caring about user experience now?

[Anonymous Coward]

Broken software running on broken hardware.

My first reaction to this fluff piece and noticing the source was "THANKS, CARLY". For it was HP that had pretty good keys to a better kingdom, but squandered them for a promise from the idiots that did the hardware half of fucking things up so wonderfully.

Re:Are sysadmins caring about user experience now?

[drinkypoo]

Go do your f'ing job and install the patches from hell, I say. And if the drop in performance bothers you, maybe we can finally talk about turning down the virus scanner to a normal level of security.

You do realize that it's the users that are always clamoring for more power? Sysadmins were happy with the 68k.

Re:Are sysadmins caring about user experience now?

[Anonymous Coward]

As a sysadmin you should know I firmly blame you and your ilk for writing crap code for the requirement to run anti virus. If you jackasses could actually program worth a damn this wouldn't be an issue. Quit fucking with the GUI and take care of your vulns looser. Hold one sec while I add another scheduled scan or two to some programmers computers.

Re:Are sysadmins caring about user experience now?

[johannesg]

ISO-9000 compliance, I believe. Having "appropriate security measures" in place is mandated.

Re:Are sysadmins caring about user experience now?

[KozmoStevnNaut]

As much as I love the connectedness of the internet, I also kind of miss the old days of DOS gaming, Win3.11, the early days of Linux and hardware without built-in backdoors ala Intel ME.

Maybe it's just nostalgia and rose-tinted glasses, but life was a lot less complicated then.

Re:Are sysadmins caring about user experience now?

[drinkypoo]

Maybe it's just nostalgia and rose-tinted glasses, but life was a lot less complicated then.

The complications were different. You had to fiddle with hypermodem or xzmodem or UUCP, you had to know the AT command set, you had to know arcane technical details of the PC ISA just to get a sound card working.

Starting up their old Sparcs OFC!

[Anonymous Coward]

I miss Sun... :-(

Re:Starting up their old Sparcs OFC!

[mdhoover]

Sparc is affected by spectre unfortunately.

We do it the easy way ...

[Anonymous Coward]

... we use systemd

It fixes everything, automagically !!

Re:We do it the easy way ...

[fisted]

True that. I mean, what's more secure than a system that can't boot /and/ resists troubleshooting?

"But fisted", you say, "Look into journalctl." [...] "Oh, you're running cyrus and the mail subsystem produces awful amounts of logs so journalctl will take minutes to even get you to the pager?" "I guess you could.. no, not that, but, maybe grep on journalctl -f?" "What do you mean, you need to see past events?" "Well don't run such a stupid mail server, it's not systemd's fault!!!" "What do you mean, it was not a problem pre systemd?" "LA LA LA I CAN'T HEAR YOU"

Re: We do it the easy way ...

[Zero__Kelvin]

You use /var/log/syslog just like with non-systemd systems. DOH!

Re: We do it the easy way ...

[Anonymous Coward]

Ever tried the --since argument to journalctl?

Clueless people

[Anonymous Coward]

The solution is simple:
1) get off of intel hardware
2) don't worry about spectre. There are no patches, and there never will be, beyond flushing branch predictors at context switch, which by now should be patched, or you could do yourself in a day or so.
3) wait 5-10 years for new silicon.
Other than that the only thing you can do is pray.
First of all, SPECTRE is almost entirely exploitable (except maybe in javascript where all scripts are run in the same process).
First you need a very specific sequence of instructions in an application that handles sensitive information.
Second you need accurate timing information on entering and exiting this sequence of instructions.
Third you need the ability to inject arbitrary data into this sequence of instructions.
Fourth: You need to know the memory layout of the program you are targeting.
Second, why do you think SPECTRE is bothering CPU designers so much?
It's because it's not easy to design speculation in a way that changes to cache are not noticed. I'm not saying it's impossible, but it requires significant redesign of the cache hierarchy. You can't fix this at just one level. Attackers could just rewrite their code to attack the next level. (if a real spectre attack is ever written)

They still have sysadmins?

[DNS-and-BIND]

I thought the sysadmin had pretty much been eliminated in favor of outsourcing IT and making the developers do it themselves. It's a prime area for cost cutting, good sysadmins aren't cheap and you won't notice they're gone because they tend to automate their jobs.

Re:They still have sysadmins?

[Anonymous Coward]

Heh, you mean the truth is out there?

As a web dev, what you say is entirely true, I tend to be the sys admin for all projects because I'm the only one hosting/developing/maintaining. That wouldn't be true if companies gave a damn to put more money into the systems that run their entire organization than they'd pay to a McDonalds fry dipping teenager, but hey that is the reality of these cheapskates. I'm honestly not sure if any of them ever actually had any money at all or if it was all just marketing for their image, but a lot of these guys you would think are completely broke the way they skimp on important business infrastructure.

Re:They still have sysadmins?

[Anonymous Coward]

You won't notice they're away immediately. Then there is an upgrade or a minor change. The automation mentioned needs a simple tweak - perhaps just a line in a script. The sysadmin could have done that as part of the upgrade - but he isn't there. Then, things come to a halt. Downtime affect many people, so much more expensive than a sysadmin's pay.

But it is instead written off as an act of god - much like a snow day. Competitors who have sysadmins don't suffer the downtime and do better - but idiot management thinks they do better because they have office landscapes or some such. Because that is all they can see and understand.

Re:They still have sysadmins?

[Anonymous Coward]

I thought the sysadmin had pretty much been eliminated in favor of outsourcing IT and making the developers do it themselves. It's a prime area for cost cutting, good sysadmins aren't cheap and you won't notice they're gone beca use they tend to automate their jobs.

"Outsourcing IT" means someone somewhere is the sysadmin. You won't notice a sysadmin missing if they've automated everything, but few automate everything; most of the time new issues or requests crop up, and the good sysadmins can use their old scripts to automate the new tasks, but without a sysadmin, good luck getting that done. If you rely on contractors to come in once a year to make changes, good luck getting them to understand everything about your computers before they make an automated system that eventually cause issues down the road. Having a dedicated sysadmin is more valuable than having a dedicated lawyer.

Performance hit

[Anonymous Coward]

Is not a rumor. And the Meltdown hit is particularly nasty if you have something that has a big cache footprint and takes long enough that calls into the kernel happens.

Re:Hosts File

[Anonymous Coward]

Er, hosts file is incredibly out dated, hell most things ignore the hosts file.

Besides this the hosts file is missing certain tools to make it viable for most things.

It needs wildcards and regex patterns to be worth a damn in todays world where you want to not just block a specific IP but rather a block of IP. Iptables generally does a much better job of this than rink a dinking around with a billion IP addresses which change like the wind blows.

I think you are the guy I see all the time going on silly rants about using a hosts file, it just isn't true, and anyone who really wants to perform blocking is better of using other methods. To anyone who ever reads this APK fellow again, hosts file is NOT the way to go, you should be looking for broader more modern tools that can actually get the job done. This man is just an insane lunatic peddling his own brand of snake oil and it will not help or protect you.

Not as critical as reported

[Anonymous Coward]

Other then the technical media blowing this up for ratings. I don't see many technical people in server administrations that concerned. Most feel nothing is in the wild we can take out time reviewing and implementing these solutions as we see fit. No sense getting all "sky is falling" mental over something that has yet to materialize. Most actual experts believe both Meltdown and Spectre can be mitigated enough to not poise much of a threat. The one "Spectre variant 2 is harder to fix but also harder to exploit. Let's move past the "sky is falling" mentality of this.

How deep do most people go down the patch hole?

[swb]

I guess what I'm referring to is digging into every single patch to try to figure out what the fuck it actually patches. And if you *do* get some kind of detail on what a specific patch actually fixes, is the information meaningful enough to decide whether you *should* apply this specific patch (relevance, risk, etc)?

Is it easier or harder now with so many vendors releasing "rollup" patches which contain multiple patches, some of which are all-inclusive and some of which require some previous rollup installed? Now picking and choosing specific patches is more or less out the door.

And then there's the question of whether the vendor even makes it easy/hard to have any control over patches, automatically just giving you patch(es) in some form or other. And of course let's not forget support -- will the vendor provide any support if you are missing patches or do you have to have them all installed anyway?

I guess what I see this boiling down to is "Who cares?" Install all the latest available patches and hope for the best. Only a full-time dedicated patch admin for a narrow product silo has the time/energy/understanding to break down the compound patching environment into something coherent and also probably is also the only one to have a complex patch management system that gives them granular control over which patches get installed and which don't.

Also, based on the last few years of software quality we're all beta testers anyway. Pretty much everything released is beta quality and hits true stability and reliability just about the point the new version is released and taming its worst initial bugs.

Badly

[Anonymous Coward]

The most overspeced systems with available patches were upgraded first to âoeget a feel for the performance impactâ some run fine. Some have been reverted because of issues unrelated to performance. Some have issues but cannot be reverted (efuses and security firmwares) a whole bunch is not yet patched due to unavailability of patches or info concerning impact.

Re: Are sysadmins caring about user experience now

[Anonymous Coward]

Probably johanne there killing the network by running kaza and LimeWire. Again.

We're Not

[Anonymous Coward]

We're all unemployed. Haven't you noticed?

Pro tip: Sysadmin is an obsolete word for DevOps.

Anyone who was working as a Sysadmin got purged for being too old.

Ask the young DevOps who took our jobs.

No patch required

[Anonymous Coward]

There is no patching required if you :
Run everything as root anyway
Run only older software
Run only one main program

Everything as root isn't an admin thing and has no separation but as soon as you run untrusted code as root, your security is non existent anyway.

Only older software : before the info on meltdown came out, no one knew, so hacks predating that are not too likely,... However, if implemented effectively, they will have gotten the keys to your kingdom and patching at that point is not enough, you'll have to reinstall...

Running only one main thing (big DB or so) on one machine: your main executable can get to all its own stuff already.

In short, if you offer cloud services to run anything, you must lock stuff down and may well be out of luck already...

Easy uodate

[Anonymous Coward]

The current release isn't stable on anything other then Skylake. Intel is releasing updates either next week or in a few more. That should patch the previous gen and hopefully doesn't crash others. That's the update I had on Thursday.

Patches are for pirates

[Anonymous Coward]

YARRRRR BITCHES

Re:Hosts File

[postbigbang]

Hmmm. Lemme see, uses a hosts file.

Ok, we'll poison arp cache. Easy enough.

If that doesn't work, we'll poison DNS.

Or, we'll code inject.

Or we'll spoof theupdates address, or see if the syslogd is correctly configured.

Hmmm, that didn't work? Let's see what ports are open. I wonder if it'll swallow out of sequence mis-formed barrages of packets.

That didn't work? We'll get a guy to install a wallwartPC with an address that's the broadcast for that segment of the VLAN.

Where there's a will, there's a way. Yes, there were lots of hideous CVEs out there before either Intel processor fault. But this one's a doozy. Ultimately, it means you're going to have to buy more hardware, it's just a matter of time. See, servers and PCs were on the decline, and well, there was some revenue that needed peaking back up again. So have a nice day, and just open up that purchase order app and quit bitchin.

Ron's experience sounds a lot like mine.

[sabbede]

Panic, patch, patch, panic, remove patches, reappy patches, panic, remove patches, deal with screaming users, patch, curse Intel....

Ryzen

[tigersha]

Our company needed a new web server in any case, so I figured the time was to take Ryzen for a spin. Very happy with it!

You have to get in 1st... apk

[Anonymous Coward]

See subject: Your methods fail on that account alone. I bypass DNS for my top 100 fav sites @ TOP of hosts (faster local resolution too & avoids all DNS' many security issues) where I spend a good near 98% of my time online. I also block the sources of your other attacks in hosts (e.g. C&C, bad servers & 3rd party malicious script sources), your 'methods' fail, badly.

* Lastly - Hosts have the ability to filter access BY PORT e.g. 1.2.3.4:3128 for example.

APK

P.S.=> Was a pleasure shooting you down EASILY "postbigbang" (behind your FAKE NAME for a FAKE LIFE online) - especially on a simple principle APK Hosts File Engine 10++ SR-1 32/64-bit https://www.google.com/search?hl=en&source=hp&biw=&bih=&q=%22APK+Hosts+File+Engine%22+and+%22start64%22&btnG=Google+Search&gbv=1/ provides... apk

Hosts files work (you're full of it)... apk

[Anonymous Coward]

See subject: I've blocked 100's of threats proven on /. alone using them. Wildcarding creates false positives. Hosts don't in specifics blocked. Regex tools aren't native to Windows & are INCREDIBLY HARD for non-programmers to manage (hosts are easy as phonebook entries to understand & edit). Most threats use host-domain names too.

WTF? Browsers do NOT ignore hosts vs. threats stupid!

(When you're REDUCED TO LIES you do ME a HUGE FAVOR - thanks!)

* The "silly guy" who 'rants on hosts' MUST be me - but it wasn't who you replied to (some butthurt fool I embarassed who is afraid to post using his registered 'lusername' as I've annihilated him before many times proven here https://yro.slashdot.org/comments.pl?sid=11782351&cid=56188765/ ) as easily as I annihilated "postbigbang" here today too https://it.slashdot.org/comments.pl?sid=11788759&cid=56189059/

APK

P.S.=> You're all the same - easy to blow away, lol... apk

Multi tiered rollout

[Archfeld]

As some have undoubtedly pointed out the true vulnerability of your system depends on exposure and the type of code run on the system, but the idea of patching only a certain segment is less than appetizing. Any business of any size has a tier of test/QA , maybe one of dev, and finally a production line. Obviously you patch one set, test/QA or dev. allow your developers to abuse the hell out of it hopefully, then roll it out to production. The company I currently work for actually has QA engineers who follow a developer provided script and pounds away at any modified branch of code ensuring all the functions do in fact function and they compare results to ensure standard replies etc. It is quite refreshing to see it done correctly for a change.

wait wait wait

[Anonymous Coward]

they are dealing with it by learning how to describe not doing anything as progress to management? that's nothing new.