Slashdot Mirror


Xen Cloud Fix Shows the Right Way To Patch Open-Source Flaws

darthcamaro writes Amazon, Rackspace and IBM have all patched their public clouds over the last several days due to a vulnerability in the Xen hypervisor. According to a new report, the Xen project was first advised of the issue two weeks ago, but instead of the knee jerk type reactions we've seen with Heartbleed and now Shellshock, the Xen project privately fixed the bug and waited until all the major Xen deployments were patched before any details were released. Isn't this the way that all open-source projects should fix security issues? And if it's not, what is?

20 of 81 comments (clear)

  1. "Gave them time" not "Waited" by martyros · · Score: 4, Funny

    The XenProject security process gives them time to patch their systems (in this case, 2 weeks). If you don't have your stuff patched by then, they won't wait for you.

    --

    TCP: Why the Internet is full of SYN.

    1. Re:"Gave them time" not "Waited" by nman64 · · Score: 4, Interesting

      Actually, the flaw in bash was also embargoed for a couple of weeks. The problem is that the original patch that was given time to circulate didn't fully fix the issue, and nobody realized that until after the embargo was lifted and the problem became public knowledge. "Responsible disclosure" was exercised in both cases, it just didn't work out well with Shellshock.

  2. But the media would lose... by charles05663 · · Score: 4, Insightful

    Their hysteria drive news cycle.

  3. Maybe? by i+kan+reed · · Score: 3, Insightful

    I mean, some open source projects don't actually have anyone doing live support and a patch happens when someone "gets around to it".

    And some exploits are out there whether you say anything or not. Slashdot users pretty regularly complain about this with bumper sticker wisdom about "security through obscurity".

    And just because the deployments are all fixed, doesn't mean someone has used that. Heartbleed(cited in the summary) was fixable within a couple days on every major linux distro with a simple update. That didn't mean no one got hacked.

    All-in-all, sure it's a good policy, but not the magic perfect, oh-lets-all-be-like-xen thing the summary makes it out to be.

    1. Re:Maybe? by kaiser423 · · Score: 3, Interesting

      It seems all pretty reasonable to me. If known exploits are out there, or if the vulnerability is known then the fix gets published right away and there's no two-week embargo. But if it appears that no one else knows about this vulnerability, then the two-week wait seems to be a great policy. Give most people that can keep their mouths shut two weeks to get everything patched up and tested.

      I get that a lot of people just chant the "security through obscurity" mantra, but obscurity really is a layer of security. It just shouldn't be your only defense. Hell, a password is a form of security through obscurity -- your salted password hash is just an obscured version of your password. So, as long as the obscurity is managed well, and in this case it appears to be, then we're good. Their document says that even small projects with no money can get on the pre-disclosure list.

    2. Re:Maybe? by i+kan+reed · · Score: 4, Informative

      your salted password hash is just an obscured version of your password.

      Negatory. Salted hashes are not reversable without a huge damned rainbow table particular to the salt, and most passwords are hashed, not encrypted.

      There isn't actually a password to recover from that.

    3. Re:Maybe? by Dishevel · · Score: 2

      I think the real idea here is that you spot an exploitable bug. You inform those who are responsible to fix it. You wait (while making sure they are working furiously on a patch.). If they are working hard toward a fix you continue to wait till it is fixed or out in the open anyway. If they blow you off and state that "they will get around to it" you let it fly.

      --
      Why is it so hard to only have politicians for a few years, then have them go away?
  4. Apples and Oranges by BenFranske · · Score: 4, Insightful

    Sure, it's an ideal situation where a bug was identified, fixed quickly and a patch pushed out and applied by large users quickly but Xen is a program which is much more centrally controlled than BASH or OpenSSL. BASH and OpenSSL are more key infrastructure bits than Xen is. What I mean is that they are integrated into FAR more devices and systems making a silent patch nearly impossible.

    1. Re:Apples and Oranges by QuietLagoon · · Score: 4, Insightful

      ... BASH and OpenSSL are more key infrastructure bits than Xen is. What I mean is that they are integrated into FAR more devices and systems making a silent patch nearly impossible.

      Quite correct.

      .
      Just try to estimate the number of devices affected by Heartbleed and Shellshock. It's probably in the billions.

      As a case in point, a single Zen installation can host hundreds, maybe even thousands, of vulnerable installations of Shellshock and Heartbleed.

      It is truly an apples and oranges comparison.

  5. It is a valid strategy by jones_supa · · Score: 2

    the Xen project privately fixed the bug and waited until all the major Xen deployments were patched before any details were released. Isn't this the way that all open-source projects should fix security issues?

    I do see value in that approach. When a vulnerability is found, it's better to report it discretely to the authors. Shouting the details to the world in the name of "openness" just causes script kiddies to go wild and nuke a bunch of machines which could have been otherwise avoided.

  6. Black hat by mwvdlee · · Score: 2

    The question is whether black hat hackers are aware of the security holes.

    Since Open Source projects communicate in the open (even if just version control commits), I find it quite likely that all major security-related projects are monitored by black hat hackers. The few weeks waiting period gives them ample time to use the security hole.

    --
    Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
    1. Re:Black hat by meustrus · · Score: 3, Interesting

      Many open source projects have specific protocols for security flaws that include having an insulated security team communicating in private with private code repositories. But even for those that don't, two weeks of security by obscurity is better than two weeks of no security at all.

      --
      I sometimes ask revealing, often ignorant-seeming questions. Maybe they're harder to answer than you think.
    2. Re:Black hat by martyros · · Score: 3, Informative

      Since Open Source projects communicate in the open (even if just version control commits), I find it quite likely that all major security-related projects are monitored by black hat hackers. The few weeks waiting period gives them ample time to use the security hole.

      That's why the Xen Project doesn't put the fix into version control until after the embargo period is over. Only people on the predisclosure list (or those able to listen in) would be able to learn about the vulnerability without doing their own audit of the code to find the bug themselves (which is very expensive).

      There's basically a balance to be struck. All users not on the predisclosure list (and thus who cannot update their systems until the embargo period is over) will continue to be privately vulnerable during the embargo period: anyone who happens to have dug deep enough and found the bug can still exploit it. But as soon as the announcement is made, everyone who hasn't yet updated is publicly vulnerable: Nobody has to search to find the bug, they just have to write an exploit for it. Being privately vulnerable is certainly bad, but being publicly vulnerable is far worse. The goal of the embargo period is to try to reduce the time that users are publicly vulnerable by extending the time they are privately vulnerable. Two weeks has been found to be a reasonable cost/benefit trade-off in our experience.

      --

      TCP: Why the Internet is full of SYN.

  7. Different audiences by meustrus · · Score: 2

    I'm sure the Xen project can keep track of all of its major players and inform them ahead of time. And nobody is a "minor player" with something so complex as Xen. It's complex enough that most users probably have support contracts with larger users. It's a lot easier to discreetly distribute a patch to that audience than to literally nearly every SSL server on the internet or every Linux user.

    --
    I sometimes ask revealing, often ignorant-seeming questions. Maybe they're harder to answer than you think.
  8. Re:Ignorance is not bliss by Chrisq · · Score: 4, Funny

    Ignorance is not bliss

    I didn't want to know that!

  9. That's how the bash issue was handled by nedlohs · · Score: 4, Informative

    That some idiot decided to publish the prenotification is just more likely when you have something in as widespread use as bash.

  10. Predisclosure should NOT be the normal practice by Hizonner · · Score: 2

    Predisclosure is very risky. You don't really know which members of your "predisclosure list" have good control over who finds out and which don't. And even with perfect control, if you're going to patch something the size of Amazon at all, you're going to have to tell a lot of people. Are you sure you want every individual who happens to have a certain job at Amazon to have the chance to exploit other people's systems?

    You're not really trusting organizations. You're trusting collections of individuals. And with that many individuals, you are going to have some bad actors. But you'd have a problem even if you could think of organizations as units with perfect policy enforcement. Suppose the NSA comes to you and says they're running a big Xen cluster (they probably are somewhere). And it's critical to national and maybe global security (it could very possibly be). Do they get on the list? How are you going to feel when they use that preannouncement to break into somebody else's system?

    Furthermore, people inferred that there was probably a Xen vulnerability from Amazon's downtime, before the official announcement. So how, exactly, was that better than having the Xen project actually announce that fact, with or without details or a patch?

    Also, it's not so easy to really know what's a "critical deployment". The fact is that, whether you're Xen or you're bash, you don't really know who's using your stuff. You don't really know what's critical. And you definitely don't know who's trustworthy.

    And all of THAT assumes that you even control the disclosure at all. If you find a problem in your software, that problem is "new to you". That does not mean that a bunch of other people don't already know about it. Especially the sort of people who make a business of exploiting these things. So you don't even know for sure who you're depriving of the knowledge.

    There's always an exception. Maybe Xen is that exception. But the idea that predisclosure should be the normal approach for software in general, whether open source or otherwise, is a very dangerous one.

  11. Grandstanding by Sheik+Yerbouti · · Score: 2

    The problem is "security researchers" want to gain notoriety so they can make more money as consultants and doing paid appearances. The way you do that is irresponsible disclosure that causes a big stir. If you tell someone I discovered heartbleed they have heard of that and will take it as a credibility indicator. If you tell them I discovered XSA-128 everyone says never heard of it. It's all about marketing and PR and making bucks. That's how these things end up with catchy names. The people doing this are acting rationally but with questionable motives and their dedication to actual security should be under great scrutiny.

  12. The problem isn't open source projects by Todd+Knarr · · Score: 2

    This is the right way to handle things, yes. The problem is that most researchers are used to dealing with proprietary software vendors whose reaction to any bug report is at best to ignore and bury the report and deny there's any problem, at worst to attack and sue the researchers. The only sane reaction to that situation is to handle things the way Heartbleed and Shellshock were: immediately publicly disclose all the details so that there are too many independent confirmations and too much publicity for the vendor to ignore the situation or deny the problem. When 99% of the time you need to follow one course, it's easy to lose track of when you're dealing with the other 1%.

  13. Xen != bash != openssl by yacc143 · · Score: 2

    Pick any random Linux box, and it will have bash/openssl installed.

    Xen on the other hand, rather specialized software, hence you have a couple of mega-users. It's easy to coordinate with a couple of professional organisations that are critically interested.

    Without such usage clusters, it's much more difficult.