Xen Cloud Fix Shows the Right Way To Patch Open-Source Flaws

darthcamaro writes Amazon, Rackspace and IBM have all patched their public clouds over the last several days due to a vulnerability in the Xen hypervisor. According to a new report, the Xen project was first advised of the issue two weeks ago, but instead of the knee jerk type reactions we've seen with Heartbleed and now Shellshock, the Xen project privately fixed the bug and waited until all the major Xen deployments were patched before any details were released. Isn't this the way that all open-source projects should fix security issues? And if it's not, what is?

Comments

"Gave them time" not "Waited"

[martyros]

The XenProject security process gives them time to patch their systems (in this case, 2 weeks). If you don't have your stuff patched by then, they won't wait for you.

Re:"Gave them time" not "Waited"

[nman64]

Actually, the flaw in bash was also embargoed for a couple of weeks. The problem is that the original patch that was given time to circulate didn't fully fix the issue, and nobody realized that until after the embargo was lifted and the problem became public knowledge. "Responsible disclosure" was exercised in both cases, it just didn't work out well with Shellshock.

But the media would lose...

[charles05663]

Their hysteria drive news cycle.

Maybe?

[i kan reed]

I mean, some open source projects don't actually have anyone doing live support and a patch happens when someone "gets around to it".

And some exploits are out there whether you say anything or not. Slashdot users pretty regularly complain about this with bumper sticker wisdom about "security through obscurity".

And just because the deployments are all fixed, doesn't mean someone has used that. Heartbleed(cited in the summary) was fixable within a couple days on every major linux distro with a simple update. That didn't mean no one got hacked.

All-in-all, sure it's a good policy, but not the magic perfect, oh-lets-all-be-like-xen thing the summary makes it out to be.

Re:Maybe?

[kaiser423]

It seems all pretty reasonable to me. If known exploits are out there, or if the vulnerability is known then the fix gets published right away and there's no two-week embargo. But if it appears that no one else knows about this vulnerability, then the two-week wait seems to be a great policy. Give most people that can keep their mouths shut two weeks to get everything patched up and tested.

I get that a lot of people just chant the "security through obscurity" mantra, but obscurity really is a layer of security. It just shouldn't be your only defense. Hell, a password is a form of security through obscurity -- your salted password hash is just an obscured version of your password. So, as long as the obscurity is managed well, and in this case it appears to be, then we're good. Their document says that even small projects with no money can get on the pre-disclosure list.

Re:Maybe?

[i kan reed]

your salted password hash is just an obscured version of your password.

Negatory. Salted hashes are not reversable without a huge damned rainbow table particular to the salt, and most passwords are hashed, not encrypted.

There isn't actually a password to recover from that.

Apples and Oranges

[BenFranske]

Sure, it's an ideal situation where a bug was identified, fixed quickly and a patch pushed out and applied by large users quickly but Xen is a program which is much more centrally controlled than BASH or OpenSSL. BASH and OpenSSL are more key infrastructure bits than Xen is. What I mean is that they are integrated into FAR more devices and systems making a silent patch nearly impossible.

Re:Apples and Oranges

[QuietLagoon]

... BASH and OpenSSL are more key infrastructure bits than Xen is. What I mean is that they are integrated into FAR more devices and systems making a silent patch nearly impossible.

Quite correct.

.
Just try to estimate the number of devices affected by Heartbleed and Shellshock. It's probably in the billions.

As a case in point, a single Zen installation can host hundreds, maybe even thousands, of vulnerable installations of Shellshock and Heartbleed.

It is truly an apples and oranges comparison.

Re:Black hat

[meustrus]

Many open source projects have specific protocols for security flaws that include having an insulated security team communicating in private with private code repositories. But even for those that don't, two weeks of security by obscurity is better than two weeks of no security at all.

Re:Ignorance is not bliss

[Chrisq]

Ignorance is not bliss

I didn't want to know that!

That's how the bash issue was handled

[nedlohs]

That some idiot decided to publish the prenotification is just more likely when you have something in as widespread use as bash.

Re:Black hat

[martyros]

Since Open Source projects communicate in the open (even if just version control commits), I find it quite likely that all major security-related projects are monitored by black hat hackers. The few weeks waiting period gives them ample time to use the security hole.

That's why the Xen Project doesn't put the fix into version control until after the embargo period is over. Only people on the predisclosure list (or those able to listen in) would be able to learn about the vulnerability without doing their own audit of the code to find the bug themselves (which is very expensive).

There's basically a balance to be struck. All users not on the predisclosure list (and thus who cannot update their systems until the embargo period is over) will continue to be privately vulnerable during the embargo period: anyone who happens to have dug deep enough and found the bug can still exploit it. But as soon as the announcement is made, everyone who hasn't yet updated is publicly vulnerable: Nobody has to search to find the bug, they just have to write an exploit for it. Being privately vulnerable is certainly bad, but being publicly vulnerable is far worse. The goal of the embargo period is to try to reduce the time that users are publicly vulnerable by extending the time they are privately vulnerable. Two weeks has been found to be a reasonable cost/benefit trade-off in our experience.