Slashdot Mirror
Debian Bug Leaves Private SSL/SSH Keys Guessable
SecurityBob writes "Debian package maintainers tend to very often modify the source code of the package they are maintaining so that it better fits into the distribution itself. However, most of the time, their changes are not sent back to upstream for validation, which might cause some tension between upstream developers and Debian packagers. Today, a critical security advisory has been released: a Debian packager modified the source code of OpenSSL back in 2006 so as to remove the seeding of OpenSSL random number generator, which in turns makes cryptographic key material generated on a Debian system guessable. The solution? Upgrade OpenSSL and re-generate all your SSH and SSL keys. This problem not only affects Debian, but also all its derivatives, such as Ubuntu." Reader RichiH also points to Debian's announcement and Ubuntu's announcement.
Who did this? You don't remove the seeding... stupid
did I mention stupid?
this is how some of the old video games were "broken" despite using "random" numbers, the seed was always the same... leading to the same sequence of events
I will not give in to the terrorists. I will not become fearful.
Ubuntu got an updated advisory at http://www.ubuntu.com/usn/usn-612-2
whether this can possibly be claimed to be an accident *dons tinfoil hat*.
But seriously, removing the code that seeds a random number generator? I can hardly imagine making such a change by accident. I may just lack a sufficiently colorful imagination, though.
(or, before resorting to conspiracy theories, we should probably ask ourselves first, "can this possibly be explained by simple stupidity?"
Every expression is true, for a given value of 'true'
This problem isn't something you can just update your system to fix. This means the basis for all remote authentication on your Debian/Ubuntu machines is compromised until you go and fix it manually.
First off I'm a big OSS supporter, yada, yada
But the point here is that the freedom that OSS gives you does require you to trust the whole distribution chain. In this case there was an added muppet who did something they shouldn't have thus rendering everything downstream insecure. OSS is great but it required great developers, given that it has take well over a year to get the advisory out it shows that the many eyes piece didn't work here, mainly because the eyes were looking at the original source not the botched packaging job.
The "easy to use" Linux box in the house uses Ubuntu and has this issue and like many people I didn't even think to check that the OpenSSL wasn't the REAL OpenSSL it was OpenSSL with muppet extensions. Maybe there needs to be some form of extension that warns that a package has been modified from its original source code and that the modification was done by "K. Frog" so you can determine whether to trust that package or look back to the source.
Or some sort of voting system on contributors (how very Web 2.0) so you can see how the people who touched your package were rated with the biggest weighting being given to the last person through the code (hand edited by Linus = 5 stars, hand edited by James Gosling = 5 stars, hand edited by the bloke who wrote clippy = 2 stars, hand edited by a bloke who removed a seed generator = 0 stars).
Having the code is great, but this makes me want to know much more about who last edited that code.
An Eye for an Eye will make the whole world blind - Gandhi
It was accidentally introduced in 2006... so that's what, another 14 years before it gets moved into 'stable'?
*grin*
It shouldn't need fixing in the first place.
Debian people screwed up. This leaves a huge distaste in my mouth for Debian (and Ubuntu).
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
note: When I have to run Linux instead of a BSD it's Debian and/or Kubuntu all the way since the benefits outweigh the negatives, but it's still an annoying habit of theirs.
Yeah, like all those times when MS cut checks for all their customers whose computers were compromised! Oh, wait...
Caveat Utilitor
I'm sure the problem will be fixed if the developers acknowledge that the problem exists. Not a big worry.
Yes, it is a big worry because any keys generated with this package are now potentially suspect. This means that anybody who's used Debian or a Debian derived distribution like Ubuntu needs to go back and destroy all host and personal keys generated since 2006. All of those keys are potentially guessable.
And that's a real vulnerability. Early versions of Netscape's SSL implementation (the first SSL implementation) were trivially crackable because of just such a vulnerability.
Need a Python, C++, Unix, Linux develop
"You fell for one of the classic blunders, the most famous being 'Never get involved in a land war in Asia' but only slightly less well known is 'Don't use poorly seeded pRNGs in cryptographic protocols!' HAHAHAHAHAHHAHAHAHAHHAHAHAHAHAHA!!!!
Test your net with Netalyzr
The seeding was removed and it wasn't noticed for TWO YEARS? In a distro as popular as Debian?
... and regenerate all the keys, yes? It isn't quite as simple as you suggest...
"All OpenSSH and X.509 keys generated on such systems must be considered untrustworthy, regardless of the system on which they are used, even after the update has been applied."
Proudly supporting the Libertarian Party.
Um.... it doesn't matter if it's been patched, you still have to regenerate all your keys.
There most certainly is something to see here, this is kind of a big deal.
http://www.random.org/analysis/dilbert.jpg
http://www.xkcd.com/221/
Downloading the patch is step one - you also need to regenerate any certificates made with OpenSSL since 2005, since they can't be guaranteed to be secure.
This has the potential to turn into a huge pain in the arse for Debian based shops, who will need to reissue SSL certificates, SSH keys, and a whole host of other essential elements of their security infrastructure.
Basic cryptographic services have been compromised for a year and your analysis is to assume on faith that it's open source so it will be fixed, so no problem?
If someone stole your crypto keys and has had them for a year...
How thoroughly might they have compromised your system by now?
How many passwords might they have stolen that you use on other systems?
What else might they have done that will give them access in the future even after you fix this?
Just regenerate your keys and no problem? The problem that guessable keys are generated will undoubtably be fixed asap, if not already. The problem that this has been the case for the last year will not be, and is a big worry.
I wake up and what do I see first thing? That there is a problem with Debian's OpenSSH package and the
Now I am thinking, "What exactly is going on here? Is choking on a bucket of cocks not a good source of randomness?"
Anyone who posts to this story saying that this is no big deal or telling other people to stop whining should simply be banned from Slashdot for life. If you cannot be bothered to read the article and you cannot be bothered to understand just what a serious vulnerability this is but you insist on insulting those who do, why should you be allowed to continue to post your ignorant bleating?
If you mod me Overrated, you are admitting that you have no penis.
In fact I'd expect that separate runs of the same program with the same parameters and environment would leave the same junk on the stack every time.
So I would hope that they have some better source of entropy than unpredictable uninitialized data of dubious randomness.
So if this is really a serious problem, then it seems to me there's already a serious problem in OpenSSH.
Uhhm, no. If you generated a key using the affected packages (ssh-keygen), as a user, your key needs to be replaced.
No, the random text your put in can be random or not. What they were talking about is a number generator that was used or applied to an algorithm that decided the prime of your key in so that the same algorithm wouldn't be used on two machines on the first then second keys. Without the random number generator, you could theoretically guess the algorithm used to generate the key based of the actions of another install and then decipher the key to gain access to whatever it was protecting.
To simplify and generalize it, if every machine uses X+1*256 to get a 256 bit key equal to 768, then you could reverse that and know X would =2 (3*256=768) and fake the credentials. The random number generator should change that to X+R*256 which make reversing the key harder because you can only solve to X+R=3 now. In practice, it will be a really larger number and a lot different process though. I can't say that I fully understand it but that simplification should show the difference well enough to give an Idea of where the problem is.
I think the point was that if the Debian maintainer had submitted his patch upstream, he'd have been told exactly how stupid it was.
Check out this bug report and tell me they weren't just being arrogant:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516
Says the first comment to the patch, in regards to what is being reversed:
"What it's doing is adding uninitialised numbers to the pool to create random numbers."
OpenSSL having more-random numbers vs. Valgrind looking good. And Valgrind won?
The consensus in the bug report seems to be not to do it, but then someone adds the patch anyways.
Not entirely true. Keys generated before the patch made it to the repos are safe - and I think quite a lot of debian boxes are old enough (I know I've got one.) There's a link in the advisory to a tool that checks if the keys are vulnerable.
This doesn't change the fact that this is a really serious fuckup. Debian lost quite some credibility in my eyes.
I, on the other hand, strongly suspect that any similar mistake at a major software corporation would in all probability be quietly ignored, if it were even noticed at all -- and if it were instead deemed enough of a public relations risk to warrant dealing with, the company would likely just silently push an update to correct the problem for future users, leaving anyone using extant keys with their arses hanging in the breeze.
But maybe I'm just being overly cynical. :-\
Cheers,
"What in the name of Fats Waller is that?"
"A four-foot prune."
Developers: We can use your help.
Schwab
Editor, A1-AAA AmeriCaptions
From this log: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516
It looks like openssl is pulling in "entropy" from uninitialized memory, causing valigrind to complain. The debian maintainer "fixed" this issue by memsetting the buffer to zero.
My question is, wouldn't I see the same behavior using grsecurity to scrub deallocated memory? From what I am seeing, this looks much more like the fault of the openssl team.
But that doesn't fix all of the machines that my public key (generated on Ubuntu) resides on that do not run Ubuntu. So yes fixing Ubuntu machines is easy, it is a PITA to have to go and re-upload my public key to all of these hosts.
Actually, here is where the Debian maintainer brought this up with upstream, and he was not told how stupid it was, in fact the Debian maintainer says, "the pool might receive less entropy." and upstream says, "If it helps with debugging, I'm in favor of removing them."
Actually, any DSA key *used* on a box with the bad SSL packages may be compromised:
From http://wiki.debian.org/SSLkeys
Additionally, some DSA keys may be compromised in the following situations:
* key generated with broken openssl = bad
* key generated with good openssl and used to ssh from a machine with bad ssl = bad
* key generated with good openssl and used to ssh from a machine with good ssl = good
This is because the random numbers used during the signature process must also be good.
From what I undestood, with my limited cryptographic knowledge and my total reluctance to dig into all the details of the sources, what happens is that when you asked your debian box to generate a key, it did so without integrating enough random data in its choice. That means that a brute force attack will not have to try every key to find yours but only a small subset, depending on the knowledge it has about your machine and its state when it generated the keys.
There is no such thing as a master key, but it would mean that several different servers may have generated the same "random" keys and that a clever attacker could create a dictionary of plausible keys (or, more plausibly, an algorithm to generate them) and try all of them on your server with a good chance to have one of them working. The dictionary would probably be huge however. A good measure would be to block an IP on SSH after a reasonnable number of failures (like 100)
The Wise adapts himself to the world. The Fool adapts the world to himself. Therefore, all progress depends on the Fool.
Especially when they won't work anymore:
The GP poster said the same thing, but this is for the benefit of other readers who were skeptical that any such policy was in place.
So, basically, once you upgrade, you'll have no apparent way to access your other machines [1] to upload your new key. That's just spiffy!
[1] Uninstall openssh-blacklist first.
Dewey, what part of this looks like authorities should be involved?
It is dangerous to be right when the government is wrong.
He would have been told, but who says he would have cared? This happens all the time, although usually it doesn't result in a giant fucking security hole. I was warning people this sort of thing would happen more than two years ago back when I was working on autopackage. It was entirely predictable.
Having distributors randomly change source code as they package it is fundamentally broken. It cannot ever work. It's not just OpenSSL that's been broken like this. Wine often went through periods in which the Debian packages were completely hosed in subtle ways. It would look like a bug in Wine, but was in fact due to parts of the software "going away", because the Debian packager felt they weren't important enough to be a part of the base install. Pointing out that their change was broken didn't help. The result was many, many hours wasted tracking down non-bugs. Then you go through a giant waste of time trying to get the bug in the packages fixed, and just as you finally get it sorted out, some other distribution introduces some other bug into their packages.
So I got sick of this, and started telling people to avoid their distros packages. Some distributors (especially Debian) took it personally and started childish slanging matches. But really, who cares how "integrated" a program is, if it could have had arbitrary bugs silently introduced? It's just a busted model of software distribution.
Imagine if Microsoft reserved the right to modify any software for Windows in any way it saw fit! Yet that's exactly what Debian (and Fedora and Mandrake and Ubuntu) said to me - they reserve the right to make any modifications they like to the software they ship, and if upstream don't like it, tough luck.
I wonder if they'll re-evaluate that policy in light of this disaster. Probably not.
It is even worse than this. Passwords that were used on a server with a weak DSA key may be compromised, as well:
From http://wiki.debian.org/SSLkeys
The problem is that the email is rather misleading, from what I understand. It talks about two places causing the problem even though only one of them actually was making valgrind complain (and you can see, purify already picked up on the problem some time before). The trick is that his patch, not included in the email, removed both calls. Which was busted. So if he'd actually sent a full patch, proposed it for inclusion and had it properly code reviewed, the mistake would probably have been caught.
No, he posted a question openssl-dev, which is a mailing list for people writing software that uses OpenSSL. This OpenSSL developer doesn't read it and that's similar with most open source project - developers often don't read mailing lists for end users.
What's more, that mail doesn't contain a patch. It contains a misleading question with two lines posted in isolation. An actual patch, submitted for an actual code review, would almost certainly have revealed the problem via context.
You don't change crypto code of all things based on an idle question on a mailing list populated mostly by users. What's next, changing the kernel scheduler based on a conversation in #kernel-newbies?
Incorrect, no matter what Ben claims. According to http://www.openssl.org/support/, openssl-dev is for the developers of openssl itself. To quote the list description: Discussions on development of the OpenSSL library. Not for application development questions!
So yes, perhaps Kurt could have been more explicit in his description of what he was trying to do, but he was definitely using the appropriate address to reach the developers.
noah
The problem is, OpenSSL needed a buffer in which to XOR several sources of randomness, including /dev/random. OpenSSL didn't bother initializing the buffer, because doing so just eats CPU cycles, and a possibly random buffer is marginally better than a non-random one. However, the Debian patch removed *all* of the randomness except for the PID, leaving the user with one of just 262148 (or so) distinct keypairs. The tool that checks if your keys have been conpromised has a list of all of those keypairs and checks if your keypair appears.
Nothing for 6-digit uids?
Don't go bragging. A lot of people know the key to rooting your wife's box.
I cann't claim to 100% understand the situation but after glancing trough the logs of the discussions and of the patches the conclusion I came to was this - OpenSSL used supposed randomness of the uninitialized memory as an added source of entropy (interesting hack, but not an example of good coding as such). Valgring caught that problem and the Debian maintainer during a cleanup fixed it. Making such a fix can be considered a preventive step against possible attack vectors by poisoning the uninitialized memory. He took it up to upstream, they did not raise red flags, but did not quite merge the âclean upâ(TM) patch either. It fell through the cracks.
The problem is that in the same file, in another function all other sources of entropy were being merged into the pool of randomness using exactly the same code line as the one code line flagged by Valgrind. The maintainer assumed that the second code line has a similar function to the first and commented that one as well. AFAIK that also did not show up in the emails to the upstream list.
So we have:
* Upstream using clever hacks that rely on uninitialized memory having some randomness to it
* Upstream using same code and same variable names to describe different things
* Upstream having no comments in the code explaining the two things above
* Maintainer slightly over-generalizing a change
* A bug slipping trough the cracks in the review processes
* Another Debian Developer discovering the bug and recognizing its significance despite all of the above
* Debian project coming out and admitting all of the above and scrambling to get fixes out to its users ASAP
I am impressed by the swift action of the people involved in fixing this. And while I think everyone can find some lesson be learned here, I think this is another good example of free software in action. And I hope that in the aftermath of this we will find ways to prevent this from happening in the future without stifling our progress.
http://www.aigarius.com/blog/2008/05/14/too-similar-to-be-different/