Slashdot Mirror
Debian Bug Leaves Private SSL/SSH Keys Guessable
SecurityBob writes "Debian package maintainers tend to very often modify the source code of the package they are maintaining so that it better fits into the distribution itself. However, most of the time, their changes are not sent back to upstream for validation, which might cause some tension between upstream developers and Debian packagers. Today, a critical security advisory has been released: a Debian packager modified the source code of OpenSSL back in 2006 so as to remove the seeding of OpenSSL random number generator, which in turns makes cryptographic key material generated on a Debian system guessable. The solution? Upgrade OpenSSL and re-generate all your SSH and SSL keys. This problem not only affects Debian, but also all its derivatives, such as Ubuntu." Reader RichiH also points to Debian's announcement and Ubuntu's announcement.
Who did this? You don't remove the seeding... stupid
did I mention stupid?
this is how some of the old video games were "broken" despite using "random" numbers, the seed was always the same... leading to the same sequence of events
I will not give in to the terrorists. I will not become fearful.
Ubuntu got an updated advisory at http://www.ubuntu.com/usn/usn-612-2
whether this can possibly be claimed to be an accident *dons tinfoil hat*.
But seriously, removing the code that seeds a random number generator? I can hardly imagine making such a change by accident. I may just lack a sufficiently colorful imagination, though.
(or, before resorting to conspiracy theories, we should probably ask ourselves first, "can this possibly be explained by simple stupidity?"
Every expression is true, for a given value of 'true'
This problem isn't something you can just update your system to fix. This means the basis for all remote authentication on your Debian/Ubuntu machines is compromised until you go and fix it manually.
Got up this morning, booted the machine and got a software update first thing: OpenSSH (et al) updates for my Ubuntu Gutsy install. Then I show up over here and see why. Presumably Feisty and Hardy got them as well - they are listed on the Ubuntu announcement.
First off I'm a big OSS supporter, yada, yada
But the point here is that the freedom that OSS gives you does require you to trust the whole distribution chain. In this case there was an added muppet who did something they shouldn't have thus rendering everything downstream insecure. OSS is great but it required great developers, given that it has take well over a year to get the advisory out it shows that the many eyes piece didn't work here, mainly because the eyes were looking at the original source not the botched packaging job.
The "easy to use" Linux box in the house uses Ubuntu and has this issue and like many people I didn't even think to check that the OpenSSL wasn't the REAL OpenSSL it was OpenSSL with muppet extensions. Maybe there needs to be some form of extension that warns that a package has been modified from its original source code and that the modification was done by "K. Frog" so you can determine whether to trust that package or look back to the source.
Or some sort of voting system on contributors (how very Web 2.0) so you can see how the people who touched your package were rated with the biggest weighting being given to the last person through the code (hand edited by Linus = 5 stars, hand edited by James Gosling = 5 stars, hand edited by the bloke who wrote clippy = 2 stars, hand edited by a bloke who removed a seed generator = 0 stars).
Having the code is great, but this makes me want to know much more about who last edited that code.
An Eye for an Eye will make the whole world blind - Gandhi
It was accidentally introduced in 2006... so that's what, another 14 years before it gets moved into 'stable'?
*grin*
It shouldn't need fixing in the first place.
Debian people screwed up. This leaves a huge distaste in my mouth for Debian (and Ubuntu).
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
note: When I have to run Linux instead of a BSD it's Debian and/or Kubuntu all the way since the benefits outweigh the negatives, but it's still an annoying habit of theirs.
http://en.wikipedia.org/wiki/Linus's_law
I understand he's not unheard of in this arena.
Yeah, like all those times when MS cut checks for all their customers whose computers were compromised! Oh, wait...
Caveat Utilitor
I'm sure the problem will be fixed if the developers acknowledge that the problem exists. Not a big worry.
Yes, it is a big worry because any keys generated with this package are now potentially suspect. This means that anybody who's used Debian or a Debian derived distribution like Ubuntu needs to go back and destroy all host and personal keys generated since 2006. All of those keys are potentially guessable.
And that's a real vulnerability. Early versions of Netscape's SSL implementation (the first SSL implementation) were trivially crackable because of just such a vulnerability.
Need a Python, C++, Unix, Linux develop
"You fell for one of the classic blunders, the most famous being 'Never get involved in a land war in Asia' but only slightly less well known is 'Don't use poorly seeded pRNGs in cryptographic protocols!' HAHAHAHAHAHHAHAHAHAHHAHAHAHAHAHA!!!!
Test your net with Netalyzr
The seeding was removed and it wasn't noticed for TWO YEARS? In a distro as popular as Debian?
... and regenerate all the keys, yes? It isn't quite as simple as you suggest...
"All OpenSSH and X.509 keys generated on such systems must be considered untrustworthy, regardless of the system on which they are used, even after the update has been applied."
Proudly supporting the Libertarian Party.
Um.... it doesn't matter if it's been patched, you still have to regenerate all your keys.
There most certainly is something to see here, this is kind of a big deal.
This has nothing to do with the vulnerability being discussed here.
http://www.random.org/analysis/dilbert.jpg
http://www.xkcd.com/221/
Downloading the patch is step one - you also need to regenerate any certificates made with OpenSSL since 2005, since they can't be guaranteed to be secure.
This has the potential to turn into a huge pain in the arse for Debian based shops, who will need to reissue SSL certificates, SSH keys, and a whole host of other essential elements of their security infrastructure.
Basic cryptographic services have been compromised for a year and your analysis is to assume on faith that it's open source so it will be fixed, so no problem?
If someone stole your crypto keys and has had them for a year...
How thoroughly might they have compromised your system by now?
How many passwords might they have stolen that you use on other systems?
What else might they have done that will give them access in the future even after you fix this?
Just regenerate your keys and no problem? The problem that guessable keys are generated will undoubtably be fixed asap, if not already. The problem that this has been the case for the last year will not be, and is a big worry.
I wake up and what do I see first thing? That there is a problem with Debian's OpenSSH package and the
Now I am thinking, "What exactly is going on here? Is choking on a bucket of cocks not a good source of randomness?"
Anyone who posts to this story saying that this is no big deal or telling other people to stop whining should simply be banned from Slashdot for life. If you cannot be bothered to read the article and you cannot be bothered to understand just what a serious vulnerability this is but you insist on insulting those who do, why should you be allowed to continue to post your ignorant bleating?
If you mod me Overrated, you are admitting that you have no penis.
In fact I'd expect that separate runs of the same program with the same parameters and environment would leave the same junk on the stack every time.
So I would hope that they have some better source of entropy than unpredictable uninitialized data of dubious randomness.
So if this is really a serious problem, then it seems to me there's already a serious problem in OpenSSH.
Uhhm, no. If you generated a key using the affected packages (ssh-keygen), as a user, your key needs to be replaced.
No, the random text your put in can be random or not. What they were talking about is a number generator that was used or applied to an algorithm that decided the prime of your key in so that the same algorithm wouldn't be used on two machines on the first then second keys. Without the random number generator, you could theoretically guess the algorithm used to generate the key based of the actions of another install and then decipher the key to gain access to whatever it was protecting.
To simplify and generalize it, if every machine uses X+1*256 to get a 256 bit key equal to 768, then you could reverse that and know X would =2 (3*256=768) and fake the credentials. The random number generator should change that to X+R*256 which make reversing the key harder because you can only solve to X+R=3 now. In practice, it will be a really larger number and a lot different process though. I can't say that I fully understand it but that simplification should show the difference well enough to give an Idea of where the problem is.
I think the point was that if the Debian maintainer had submitted his patch upstream, he'd have been told exactly how stupid it was.
Check out this bug report and tell me they weren't just being arrogant:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516
Says the first comment to the patch, in regards to what is being reversed:
"What it's doing is adding uninitialised numbers to the pool to create random numbers."
OpenSSL having more-random numbers vs. Valgrind looking good. And Valgrind won?
The consensus in the bug report seems to be not to do it, but then someone adds the patch anyways.
Not entirely true. Keys generated before the patch made it to the repos are safe - and I think quite a lot of debian boxes are old enough (I know I've got one.) There's a link in the advisory to a tool that checks if the keys are vulnerable.
This doesn't change the fact that this is a really serious fuckup. Debian lost quite some credibility in my eyes.
I, on the other hand, strongly suspect that any similar mistake at a major software corporation would in all probability be quietly ignored, if it were even noticed at all -- and if it were instead deemed enough of a public relations risk to warrant dealing with, the company would likely just silently push an update to correct the problem for future users, leaving anyone using extant keys with their arses hanging in the breeze.
But maybe I'm just being overly cynical. :-\
Cheers,
"What in the name of Fats Waller is that?"
"A four-foot prune."
Hello troll,
You have no recourse unless you have PAID to have one, like with almost everything else in the world. The GPL specifically states that it's not a warranty or guarantee that things will work.
However, if you were using it in a business, I would hope that you either a) hired your own programmers to work with the code or b) bought a support package/liability clause from someone like Red Hat. In which case it would be down to your programmers or Red Hat respectively. But we're not talking about Red Hat. Or any of the other big-name, support-contract-and-some-sort-of-indemnity-clause-included distros. They don't have this problem, presumably because they are not that stupid.
We're talking about Debian. Got a support contract / liability agreement with Debian? No? Bad luck. It's a bit like asking Microsoft to accept responsibility for your pirate copy of Windows, then, isn't it?
First off, we were not told any such thing -- or at least I've never run across any such claims myself. Secondly, this would actually seem to substantiate the many-eyes theory, as the bug was found and summarily corrected, rather than never found, or found and swept under the rug, or found first by black-hats and exploited, all of which seem quite common in the proprietary software world.
The many-eyes model does not guarantee zero bugs, nor does it guarantee speedy bug hunting. However, it would seem to guarantee that bugs will be found eventually, and also that they will be dealt with in some productive way once they are found. This model aims for the opposite of security through obscurity, that hoary old chestnut of proprietary development, by instead ensuring security by knowing exactly what things do. That also means that folks need to go through all changes -- i.e., many eyes actually need to go over the code. Open source ensures that this is possible, but actual people still need to put in the time and effort.
Cheers,
"What in the name of Fats Waller is that?"
"A four-foot prune."
On ubuntu.... according to the vulnerability description.. Once the update is applied, weak user keys will be automatically rejected where possible (though they cannot be detected in all cases). If you are using such keys for user authentication, they will immediately stop working and will need to be replaced (see step 3). OpenSSH host keys can be automatically regenerated when the OpenSSH security update is applied. The update will prompt for confirmation before taking this step.
Developers: We can use your help.
Schwab
Editor, A1-AAA AmeriCaptions
From this log: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516
It looks like openssl is pulling in "entropy" from uninitialized memory, causing valigrind to complain. The debian maintainer "fixed" this issue by memsetting the buffer to zero.
My question is, wouldn't I see the same behavior using grsecurity to scrub deallocated memory? From what I am seeing, this looks much more like the fault of the openssl team.
It's not arrogance. Or it's at least not arrogance alone. As a distro maintainer I can tell you that upstream providers generally do not care about distro-specific patches. Even if you have a patch that would be useful on more than on distro, there's often a reluctance to inherit code to support any installation method other than source tarballs.
That's not to say that upstream providers are unduly arrogant either -- if you're happy with the way your build/install system works, why would you want to maintain patches for some other system that you don't even use? It's extra code to keep up, and requires more testing on more platforms, and it it doesn't make your core package any better.
But that doesn't fix all of the machines that my public key (generated on Ubuntu) resides on that do not run Ubuntu. So yes fixing Ubuntu machines is easy, it is a PITA to have to go and re-upload my public key to all of these hosts.
Actually, here is where the Debian maintainer brought this up with upstream, and he was not told how stupid it was, in fact the Debian maintainer says, "the pool might receive less entropy." and upstream says, "If it helps with debugging, I'm in favor of removing them."
Actually, any DSA key *used* on a box with the bad SSL packages may be compromised:
From http://wiki.debian.org/SSLkeys
Additionally, some DSA keys may be compromised in the following situations:
* key generated with broken openssl = bad
* key generated with good openssl and used to ssh from a machine with bad ssl = bad
* key generated with good openssl and used to ssh from a machine with good ssl = good
This is because the random numbers used during the signature process must also be good.
From what I undestood, with my limited cryptographic knowledge and my total reluctance to dig into all the details of the sources, what happens is that when you asked your debian box to generate a key, it did so without integrating enough random data in its choice. That means that a brute force attack will not have to try every key to find yours but only a small subset, depending on the knowledge it has about your machine and its state when it generated the keys.
There is no such thing as a master key, but it would mean that several different servers may have generated the same "random" keys and that a clever attacker could create a dictionary of plausible keys (or, more plausibly, an algorithm to generate them) and try all of them on your server with a good chance to have one of them working. The dictionary would probably be huge however. A good measure would be to block an IP on SSH after a reasonnable number of failures (like 100)
The Wise adapts himself to the world. The Fool adapts the world to himself. Therefore, all progress depends on the Fool.
Especially when they won't work anymore:
The GP poster said the same thing, but this is for the benefit of other readers who were skeptical that any such policy was in place.
So, basically, once you upgrade, you'll have no apparent way to access your other machines [1] to upload your new key. That's just spiffy!
[1] Uninstall openssh-blacklist first.
Dewey, what part of this looks like authorities should be involved?
It is dangerous to be right when the government is wrong.
He would have been told, but who says he would have cared? This happens all the time, although usually it doesn't result in a giant fucking security hole. I was warning people this sort of thing would happen more than two years ago back when I was working on autopackage. It was entirely predictable.
Having distributors randomly change source code as they package it is fundamentally broken. It cannot ever work. It's not just OpenSSL that's been broken like this. Wine often went through periods in which the Debian packages were completely hosed in subtle ways. It would look like a bug in Wine, but was in fact due to parts of the software "going away", because the Debian packager felt they weren't important enough to be a part of the base install. Pointing out that their change was broken didn't help. The result was many, many hours wasted tracking down non-bugs. Then you go through a giant waste of time trying to get the bug in the packages fixed, and just as you finally get it sorted out, some other distribution introduces some other bug into their packages.
So I got sick of this, and started telling people to avoid their distros packages. Some distributors (especially Debian) took it personally and started childish slanging matches. But really, who cares how "integrated" a program is, if it could have had arbitrary bugs silently introduced? It's just a busted model of software distribution.
Imagine if Microsoft reserved the right to modify any software for Windows in any way it saw fit! Yet that's exactly what Debian (and Fedora and Mandrake and Ubuntu) said to me - they reserve the right to make any modifications they like to the software they ship, and if upstream don't like it, tough luck.
I wonder if they'll re-evaluate that policy in light of this disaster. Probably not.
Most users do not generate nor use ssh/ssl keys or certificates.
The real questions are
vi +
Found this post at openssl-dev list by Kurt Roeckx ( AFAIK the Debian OpenSSL Team member that made this RNG-clean patch )
http://www.mail-archive.com/openssl-dev@openssl.org/msg21156.html
Extract:
"What I currently see as best option is to actually comment out those 2 lines of code. But I have no idea what effect this really has on the RNG. The only effect I see is that the pool might receive less entropy. But on the other hand, I'm not even sure how much entropy some unitialised data has.
What do you people think about removing those 2 lines of code?
Kurt
"
BTW, i thought that Debian had some kind of policies about testing each package before committing changes in testing/stable branches. Also, the following paper, contributed by another poster, says interesting things about touching cryptographic code, we have to learn from this experience and have tighter policies !
" In a narrow sense, the security flaw we found in the Netscape browser serves merely as an anecdote to emphasize the difficulty of generating cryptographically strong random numbers. But there's a broader moral to the story. The security community has painfully learned that small bugs in a security-critical module of a software system can have serious consequences, and that such errors are easy to commit. The only way to catch these mistakes is to expose the source code to scrutiny by security experts.
Peer review is essential to the development of any secure software. Netscape did not encourage outside auditing or peer review of its software-and that goes against everything the security industry has learned from past mistakes. By extension, without peer review and intense outside scrutiny of Netscape's software at the source-code level, there is simply no way consumers can know where there will be future security problems with Netscape's products. "
http://www.cs.berkeley.edu/~daw/papers/ddj-netscape.html"
It is even worse than this. Passwords that were used on a server with a weak DSA key may be compromised, as well:
From http://wiki.debian.org/SSLkeys
The problem is that the email is rather misleading, from what I understand. It talks about two places causing the problem even though only one of them actually was making valgrind complain (and you can see, purify already picked up on the problem some time before). The trick is that his patch, not included in the email, removed both calls. Which was busted. So if he'd actually sent a full patch, proposed it for inclusion and had it properly code reviewed, the mistake would probably have been caught.
Or a bit like asking Microsoft to accept responsibility for the copy of Windows that you paid for.
Look through the Windows license agreement some time and see how many times phrases like "THERE IS NO WARRANTY OR CONDITION OF ANY KIND", "YOU ARE NOT ENTITLED TO ANY DAMAGES" and "IN NO EVENT SHALL MICROSOFT OR ITS SUPPLIERS BE LIABLE" show up.
The only difference with Microsoft is that if Windows breaks you don't get to keep both pieces.
Ben Laurie of OpenSSL/Apache puts it into some context:
http://www.links.org/?p=327#comment-176642
Obviously some of the OpenSSL devs probably should've been like "zOMG, SITUATION FUBAR", but it wasn't a formal code review being requested, more of a "hey, what do you think of this and this?" and the patch was never submitted to upstream.
You will want to download this tool to check each system:
http://security.debian.org/project/extra/dowkd/dowkd.pl.gz
To use it:
$ perl dowkd.pl user
$ perl dowkd.pl host localhost
See http://lists.debian.org/debian-security-announce/2008/msg00152.html
Why isn't such a scan isn't part of the package update? http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=481106
No, he posted a question openssl-dev, which is a mailing list for people writing software that uses OpenSSL. This OpenSSL developer doesn't read it and that's similar with most open source project - developers often don't read mailing lists for end users.
What's more, that mail doesn't contain a patch. It contains a misleading question with two lines posted in isolation. An actual patch, submitted for an actual code review, would almost certainly have revealed the problem via context.
You don't change crypto code of all things based on an idle question on a mailing list populated mostly by users. What's next, changing the kernel scheduler based on a conversation in #kernel-newbies?
Incorrect, no matter what Ben claims. According to http://www.openssl.org/support/, openssl-dev is for the developers of openssl itself. To quote the list description: Discussions on development of the OpenSSL library. Not for application development questions!
So yes, perhaps Kurt could have been more explicit in his description of what he was trying to do, but he was definitely using the appropriate address to reach the developers.
noah
The problem is, OpenSSL needed a buffer in which to XOR several sources of randomness, including /dev/random. OpenSSL didn't bother initializing the buffer, because doing so just eats CPU cycles, and a possibly random buffer is marginally better than a non-random one. However, the Debian patch removed *all* of the randomness except for the PID, leaving the user with one of just 262148 (or so) distinct keypairs. The tool that checks if your keys have been conpromised has a list of all of those keypairs and checks if your keypair appears.
Nothing for 6-digit uids?
This has been explained elsethread, but just in case anyone wanders into this one....
The problem is not the fact that the uninitialized memory seeding was removed. The problem is that the removal was done in an incompetent and destructive manner. Rather than remove only the uninitialized memory seeding, the maintainer managed to remove all seeding.
Reading uninitialized memory as part of random seeding isn't very useful but it's not bad either, since at worst the data is predictable, and OpenSSL mixes it with lots of other randomness. But when you botch your change and completely destroy any randomness present in the system, this is not the fault of the software you were trying to "fix".
If you mod me Overrated, you are admitting that you have no penis.
I am going to hijack this thread because I have an eCommerce site hosted on Debian and this made me nervous.
According to the reps at Thawte, if you are using a third party ssl cert (thawte, verisign, etc), this does not affect you. According to them, this is only dangerous for people who have generated their own SSL certs for their sites from the ground up (probably a minority, I would guess). If anybody has any information to the contrary, please let me know.
weirdest thing I ever saw: scientology advertising on slashdot.
Don't go bragging. A lot of people know the key to rooting your wife's box.
I cann't claim to 100% understand the situation but after glancing trough the logs of the discussions and of the patches the conclusion I came to was this - OpenSSL used supposed randomness of the uninitialized memory as an added source of entropy (interesting hack, but not an example of good coding as such). Valgring caught that problem and the Debian maintainer during a cleanup fixed it. Making such a fix can be considered a preventive step against possible attack vectors by poisoning the uninitialized memory. He took it up to upstream, they did not raise red flags, but did not quite merge the âclean upâ(TM) patch either. It fell through the cracks.
The problem is that in the same file, in another function all other sources of entropy were being merged into the pool of randomness using exactly the same code line as the one code line flagged by Valgrind. The maintainer assumed that the second code line has a similar function to the first and commented that one as well. AFAIK that also did not show up in the emails to the upstream list.
So we have:
* Upstream using clever hacks that rely on uninitialized memory having some randomness to it
* Upstream using same code and same variable names to describe different things
* Upstream having no comments in the code explaining the two things above
* Maintainer slightly over-generalizing a change
* A bug slipping trough the cracks in the review processes
* Another Debian Developer discovering the bug and recognizing its significance despite all of the above
* Debian project coming out and admitting all of the above and scrambling to get fixes out to its users ASAP
I am impressed by the swift action of the people involved in fixing this. And while I think everyone can find some lesson be learned here, I think this is another good example of free software in action. And I hope that in the aftermath of this we will find ways to prevent this from happening in the future without stifling our progress.
http://www.aigarius.com/blog/2008/05/14/too-similar-to-be-different/