Slashdot Mirror
Xen Cloud Fix Shows the Right Way To Patch Open-Source Flaws
darthcamaro writes Amazon, Rackspace and IBM have all patched their public clouds over the last several days due to a vulnerability in the Xen hypervisor. According to a new report, the Xen project was first advised of the issue two weeks ago, but instead of the knee jerk type reactions we've seen with Heartbleed and now Shellshock, the Xen project privately fixed the bug and waited until all the major Xen deployments were patched before any details were released. Isn't this the way that all open-source projects should fix security issues? And if it's not, what is?
The XenProject security process gives them time to patch their systems (in this case, 2 weeks). If you don't have your stuff patched by then, they won't wait for you.
TCP: Why the Internet is full of SYN.
Their hysteria drive news cycle.
I mean, some open source projects don't actually have anyone doing live support and a patch happens when someone "gets around to it".
And some exploits are out there whether you say anything or not. Slashdot users pretty regularly complain about this with bumper sticker wisdom about "security through obscurity".
And just because the deployments are all fixed, doesn't mean someone has used that. Heartbleed(cited in the summary) was fixable within a couple days on every major linux distro with a simple update. That didn't mean no one got hacked.
All-in-all, sure it's a good policy, but not the magic perfect, oh-lets-all-be-like-xen thing the summary makes it out to be.
So you can have security and patches before the attackers, if you pay enough?
Seems like a model I already don't like.
You mean waiting until all the companies with support contracts with them were fixed. If you pay the supplier for support then it's in their interest to patch you up asap. If you don't pay then you're on your own.
Sure, it's an ideal situation where a bug was identified, fixed quickly and a patch pushed out and applied by large users quickly but Xen is a program which is much more centrally controlled than BASH or OpenSSL. BASH and OpenSSL are more key infrastructure bits than Xen is. What I mean is that they are integrated into FAR more devices and systems making a silent patch nearly impossible.
In a completely utiopian environment void of ego's and money making potentials and well funded state exploitation.
A 0-day vuln today can give you a leg up within the security research arena leading to future income potential, or make you immediate cash on the black market of criminal viri & malware or be purchased by the highest bidding state/government organization(s).
Did I forget anything is this a purely rehtorical question?
the Xen project privately fixed the bug and waited until all the major Xen deployments were patched before any details were released. Isn't this the way that all open-source projects should fix security issues?
I do see value in that approach. When a vulnerability is found, it's better to report it discretely to the authors. Shouting the details to the world in the name of "openness" just causes script kiddies to go wild and nuke a bunch of machines which could have been otherwise avoided.
The question is whether black hat hackers are aware of the security holes.
Since Open Source projects communicate in the open (even if just version control commits), I find it quite likely that all major security-related projects are monitored by black hat hackers. The few weeks waiting period gives them ample time to use the security hole.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
I'm sure the Xen project can keep track of all of its major players and inform them ahead of time. And nobody is a "minor player" with something so complex as Xen. It's complex enough that most users probably have support contracts with larger users. It's a lot easier to discreetly distribute a patch to that audience than to literally nearly every SSL server on the internet or every Linux user.
I sometimes ask revealing, often ignorant-seeming questions. Maybe they're harder to answer than you think.
And what exactly happened differently in shellshock's case?
Who writes these posts anyway? A freaking 13 year old with reading impairment?
Ignorance is not bliss
That works well for the major players, but how does it improve the situation of the smaller companies having their own deployments?
No, you don't need to pay to be on pre-disclosure list.
http://www.xenproject.org/security-policy.html
Perhaps there isn't a single universal best police in regards to this. What works best in one situation is not necessarily what works best in all of them.
This is my signature. There are many like it, but this one is mine.
That some idiot decided to publish the prenotification is just more likely when you have something in as widespread use as bash.
So unless you lived under a rock, most people knew there was a security bug out there (why else would the big cloud providers be forcing a restart of all my VMs?) We didn't know what it was, and because I'm not part of the preferred client group my servers didn't get patched prior to disclosure. So for me, no this isn't sufficient. I prefer the more open way of doing it, versus fixing it in a closed "preferred client" way that they handled this.
Predisclosure is very risky. You don't really know which members of your "predisclosure list" have good control over who finds out and which don't. And even with perfect control, if you're going to patch something the size of Amazon at all, you're going to have to tell a lot of people. Are you sure you want every individual who happens to have a certain job at Amazon to have the chance to exploit other people's systems?
You're not really trusting organizations. You're trusting collections of individuals. And with that many individuals, you are going to have some bad actors. But you'd have a problem even if you could think of organizations as units with perfect policy enforcement. Suppose the NSA comes to you and says they're running a big Xen cluster (they probably are somewhere). And it's critical to national and maybe global security (it could very possibly be). Do they get on the list? How are you going to feel when they use that preannouncement to break into somebody else's system?
Furthermore, people inferred that there was probably a Xen vulnerability from Amazon's downtime, before the official announcement. So how, exactly, was that better than having the Xen project actually announce that fact, with or without details or a patch?
Also, it's not so easy to really know what's a "critical deployment". The fact is that, whether you're Xen or you're bash, you don't really know who's using your stuff. You don't really know what's critical. And you definitely don't know who's trustworthy.
And all of THAT assumes that you even control the disclosure at all. If you find a problem in your software, that problem is "new to you". That does not mean that a bunch of other people don't already know about it. Especially the sort of people who make a business of exploiting these things. So you don't even know for sure who you're depriving of the knowledge.
There's always an exception. Maybe Xen is that exception. But the idea that predisclosure should be the normal approach for software in general, whether open source or otherwise, is a very dangerous one.
> how does it improve the situation of the smaller companies
By the time the issue becomes public, Red Hat will already have a good update ready for you, so you can just "yum update xen" and you're good to go.
Contrast this with shellshocker, where after the issue was in the media, Red Hat started trying to figure out how to handle it, and over the next several days they put multiple patches trying to address it in different ways, while upstream bash went in a completely different direction. The bash team is still trying to decide exactly how the underlying issues should be handled on a permanent basis, and we'll all be vulnerable for at least a few more days while they figure that out.
Do you remember about two years ago when there was that vulnerability that allowed bad actors is _easily_ take down DNS servers? Sending one wrong packet to the wikipedia DNS servers would take down wikipedia. You probably don't remember that, because it didn't get any press. The reason why is that after I discovered it, I contacted the project maintainers and over the next 24 hours a fix was developed. The following day, the fix was applied to high profile targets like Wikipedia. On the third day, packagers including Red Hat and FreeBSD got update packages ready and sent them out for anyone who auto-updates, along with a notice on the mailing list of users. Only after the patched binary was available through standard packages were the details of the attack public. Therefore, few people were affected it t wasn't news.
so, only the big established players with insider connections get access to the "fast lanes" of fixing known vulnerabilities.... sound familiar?
this isn't the right way to push the fix at all. an outsider found the exploit, so the exploit is already public... release the info and release the patch immediately.
just because you're in bed with big players doesn't mean you get to treat the rest of us like shit.
FUCK XEN. switching to OpenVZ immediately.
The problem is "security researchers" want to gain notoriety so they can make more money as consultants and doing paid appearances. The way you do that is irresponsible disclosure that causes a big stir. If you tell someone I discovered heartbleed they have heard of that and will take it as a credibility indicator. If you tell them I discovered XSA-128 everyone says never heard of it. It's all about marketing and PR and making bucks. That's how these things end up with catchy names. The people doing this are acting rationally but with questionable motives and their dedication to actual security should be under great scrutiny.
So this situation really was handled with aplomb. However, saying that we "should" handle things this way is about as dangerous as saying we "should" shout out the details of every vulnerability we find. Keeping things internal prevents the community from stepping up. I doubt that all the folks who have dealt with heartbleed were involved in SSL beforehand. But they were helpful because they knew they were needed, and their ignorance would have hurt us badly. On the other hand, shouting everything out feels like a dumb thing to do. So instead of some off-the-cuff polarizing question like "shouldn't we always handle things this way for EVERZ" is precisely the wrong response. Its actually the very wrongest.
Discretion, intuition, and rapid initiative. That is how we "should" handle these things. The specifics are case by case.
Well said.
If vendors want to withhold detailed notification until a patch is available I don't much care.
However withholding patches until after a "chosen" subset of user community has casually had the opportunity to fix their shit is great for you if your chosen and worse if your everyone else.
Better to announce in advance on such a date at such a time patch of import x shall be released to all concerned... let everyone who cares plan their maintenance windows accordingly.
This is the right way to handle things, yes. The problem is that most researchers are used to dealing with proprietary software vendors whose reaction to any bug report is at best to ignore and bury the report and deny there's any problem, at worst to attack and sue the researchers. The only sane reaction to that situation is to handle things the way Heartbleed and Shellshock were: immediately publicly disclose all the details so that there are too many independent confirmations and too much publicity for the vendor to ignore the situation or deny the problem. When 99% of the time you need to follow one course, it's easy to lose track of when you're dealing with the other 1%.
That's a terrible way to do it. You protect the big guys, but when the exploit info is finally released, any compnay not lucky enough to be notified is screwed with a zeroday. Its making it safer for customers to use the estabished providers, entrenching their market share.
If there were a comerical partner backin a project that knows of a critical flaw, they should give equal acess to all their customers with support contracts. If I was a citrix/xen customer I'd be pissed.
can we find and name that idiot?
Defense is depth matters, many network exploitable vulnerabilities can be mitigated before they ever get to the server gear. In case of bash only some cases/methods can be blocked. This does mean in many cases it's more than just the authors that need to know.
No sir I dont like it.
i know find out there is a back door and that millions might be vulnerable and DONT TELL ANYONE thus increasing likely hood someone is damaged...i don't know if there are cases of lawsuits like this or over it but there ought to be.
in there case it would have bene awful funny if they get hacked to pieces a week after knowing.
keeping the pool of minds small = less chance and more time to fix.
Pick any random Linux box, and it will have bash/openssl installed.
Xen on the other hand, rather specialized software, hence you have a couple of mega-users. It's easy to coordinate with a couple of professional organisations that are critically interested.
Without such usage clusters, it's much more difficult.
Your reply makes no sense and has no logic. I highly doubt if you ever read the Xen security policy page.
Firstly, it's not only "big established players" are allowed to be on pre-disclosure list. If you're a legitimate user you can be on that list.
Secondly, if a security bug is publicly discussed, Xen project will not enforce embargo period, instead patch is made immediately available to public (there has been several instances). In this XSA-108 case, the vulnerability was not public, so there was embargo period.
Thirdly, how did you come to your conclusion? Does OpenVZ have a better policy that attracts you so much? If so, I would like to hear about it.
What is comes down to is agility and design. You don't control other peoples knowledge and communications to mitigate your flaws, you ensure that you have a process for managing them effectively if you care. If a service is that valueable to you ensure that you have two service delivery mechanisms. Yes its expensive but you've just said that this channel is that important to you. Or live with the risk and keep the cash, in both cases the ball is in your court.
The whole censorship approach is created by anal retentive morons who don't understand risk, the mathematics of access control or the environment in which they're building systems.
I really see these exploits and publishing their existence as a double edged sword problem. On one hand, as an asministrator of systems I'd like to know immediately that there is an exploit so I can shut down a service, if I can, so it doesn't get exploited before I can patch it. On the other hand if the knowledge of an exploit gets into the public then more people are going to try to exploit systems that may be vulnerable, so keeping it quiet may also be beneficial.
Where do you draw the line, and for what reasons? Heartbleed was a major exploit, but was easily patched in short order, however, how many systems had been exploited before the patch was released and by whom? The same goes for Shellshock, although that exploit wasn't patched before it went public and we still see exploits at work due to bash's abundance.
I lean toward the make-it-publicly-known-ASAP camp as I would want to be able to take proactive measures to mitigate any vulnerability before it was exploited or patched. In some cases (not necessarily the Heartbleed and Shellshock examples) patching may fix the vulnerability, but the system may still be compromised if it has been exploited and the damage is already done; and amy continue until you rebuild the system. If the vulnerability was kept quiet during patch development, not knowing gave the attacker time to get what ever they were after off a compromised system before the patch, and potentially not stop further data theft after the patch.
It's really hard to say which option--to tell or not to tell--is the best course. When a vulnerability is found by a security researcher doesn't necessarily correlate to no infected systems in the wild. Who knows who found it first, a good guy or a bad guy, and there may very well be systems exploited in the wild that a patch has little effect on other than fixing a hole that's already been plumbed out by a bad guy.
Pre-disclosure and a quarantine period is useful for both infrastructure security bugs like xen and widely deployed security bugs like bash and openssl.
In the case of an infrastructure bug the quarantine period allows the service providers a chance to patch and restart their services before their customers become vulnerable.
In the case of a widely deployed security bug it give the OS/software vendors the chance to investigate the bug. Create patches and test/verify their patches and pre-stage them so when the bug is disclosed publicly sites that do automatic updates are already patched.
I think the two week window used by xen is a good window balancing giving time for people to fix the problem but limiting the time between discovery and public knowledge/exploit.