Debian Project Servers Compromised

Sean was one of many to pass along the bad news from the debian-announce mailing list: "Some Debian Project machines have been compromised. This is a very unfortunate incident to report about. Some Debian servers were found to have been compromised in the last 24 hours. The archive is not affected by this compromise! In particular the following machines have been affected: 'master' (Bug Tracking System), 'murphy' (mailing lists), 'gluck' (web, cvs), 'klecker' (security, non-us, web search, www-master). Some of these services are currently not available as the machines undergo close inspection. Some services have been moved to other machines (www.debian.org for example). The security archive will be verified from trusted sources before it will become available again." They were going to announce 3.0r2 this morning; they've checked it and it's unaffected but obviously they're still postponing that release.

Comments

apt

[isorox]

Of course this raises the whole issue of apt-get. We all rely on apt-get update && apt-get upgrade, all it takes is someone to compromise the servers and insert a backdoor

Re:apt

[Anonymous Coward]

apt-secure uses strong cryptographic methods to verify the authenticity of packages in the archive. It may be the default apt-get for sarge, depending on man-power issues.

Digital Signing of Packages?

[Chris_Jefferson]

This is the second time this has happened to a big open-source project (the first being the GNU servers a while ago). All packages by both groups are "md5" signed, which is supposed to protect against malicous hacking. However if the root server is comprimised, this doesn't help. Companies (including at least Microsoft, and the people who make ad-aware) who distribute files over the internet sign them with an RSA (or similar) key, and the computer which does this signing is kept disconnected from the internet. For such large projects which are installed by millions of people, might a similar system not be a good idea?

Re:Digital Signing of Packages?

[tfheen]

The Packages files includes md5 sums of all the .debs, the Release file contains the md5 sum of all the Packages files, and the Release file itself is signed using GPG. Using apt-check-sigs you can automate the checking of the packages you are installing.

Re:Digital Signing of Packages?

[naitro]

Consider this. A debian developer's workstation at home is compromised, and the attacker installs a keylogger. What would stop the attacker from creating an approved package and then upload it into the repository?

Now what's that they say about chains and the weakest link?

Re:Digital Signing of Packages?

[Flower]

Let's see...

1. It's possible that the developer would keep track of his commits and know he most certainly didn't submit that patch at 02:00 while he was out drinking.
   
2. The sysadmin keeps noticing that silly log saying Developer X who only has rights to commit to the X11 stuff keeps trying to commit a kernel patch.
   
3. The 70 year old neighbor who has nothing better to do than watch the neighborhood dials 911 when somebody starts poking around the developers house.
   
4. "Attacker, meet Fluffy my faithful, full-grown mastiff. Fluffy, eat attacker."
   
5. Security system
   
6. The fact that we're talking Debian here and not RH or SuSE. The amount of risk and resources it would take to Mission Impossible this poor guy's house, wait until we know we have his key in our logger and then M.I. his house again isn't worth the investment.
   

Now what's that they say about chains and the weakest link?

That you need to do a little more research before you can write that piece of fiction and become the next Tom Clancy.

Signatures?

[Sits]

Are deb's signed? (I'm not that familiar with debian but I'd imagine they are) If so then just tell apt-get to not install debs that don't match a known signature...

Where's the confirmation from debian people?

[mackstann]

I've seen no confirmation of this by anyone @debian.org. So what's the deal? Real or not?

There was some fuss on the debian-user list, and this was labeled a hoax, yet I saw no official word that it was true.

Re:Where's the confirmation from debian people?

[Raphael]

Thanks for confirming this. Unfortunately, the way you confirmed it is very dangerous.

Your message contains:

• no date
• no precise reference to the report that you are confirming

So from now one, your "confirmation" can be used by anybody who wants to claim that some random report of theirs is "confirmed by a debian developer". Until you revoke your own key, of course. That's a pity.

Re:...not the archive.

[nchip]

The server that pushes .debs to archive is running debian/sparc (donated by sun btw), so probably the cracker didn't know how to port his leet exploit to sparc (all the comprimised machines were 1386).

Everything's a tradeoff

[buddha42]

On the one hand stuff like this scare's the hell out of me, but on the other hand I'm very reasurred by how the debian community handles it. Full disclosure, detailed explanations, and very conservative thinking (exibited by the "3.0r2 is fine, but we're not releasing it anyway just to be anally sure").

At this point I would like to see the debian team develop some written policies and procedures for how they intend to prevent this sort of thing in the future. I checked the site and while there's security info for how to secure your box, there's no policies on 'how does the debian project secure itself'.

Lastly, one concept you have to keep in mind, we have no idea how often other OS's key servers are cracked because they'd never tell us.

Re:OpenBSD

[psamuels]

If Debian ran OpenBSD, this wouldn't have happened!

OpenBSD prevents stolen passwords from being used to log into a system? How?

Re:How in the world...

[Jade E. 2]

...it was due to a password compromise.

That doesn't really make it any better. That means that either a) root (or a highly privileged user) had the same password on 4 important machines, or b) there's a local root exploit in the software they're running. Neither possibility makes me feel warm and fuzzy about using their software again...

Of course, we shouldn't jump to conclusions until we get more information, but really, I don't see an easy way out of this.

So what do we do to prevent this in the future?

[finkployd]

First GNU, then Bitkeeper, now this, whatever shall we do?

Simple, the technology has existed for decades now.

A little something I like to call "Public Key Cryptography"

With this "Public Key Cryptography" you could conceivably sign software in such a way that it could not be altered without breaking the signature, AND ensure that nobody else could forge this digital signature (you are keeping your private key private right?)

MD5 Hashes are a step in the right direction, but by themselves are meaningless. Sort of like improving your home's security by drilling holes in your door to mount a deadbolt but not actually taking the final step and INSTALLING THE DEADBOLT.

So let's take these MD5 hashes and encrypt them with the package maintainer's private key (or distribution maintainer, whatever). Then dpkg (or rpm, emerge, whatever your favorite package tool is) could be written to decrypt this hash with the corresponding public key. Wait, there is more! Then it could generate it's own MD5 hash of the package in question and COMPARE it to the decrypted hash it just created. If they match, the package is unaltered AND came from a trusted source. This my friends is what we like to call a "digital signature"

I don't care how you do it, GPG, x.509, whatever. I'm actually leaning toward x.509 since it seems to me to make more sense to have the distro maintainer run his/her own CA and issue certs to package maintainers. This CA could then be included in whatever package tool is used and viola. No mucking about with the web 'o trust (Which rocks for ad hoc trust relationships like between people emailing each other, but sucks for this kind of hierarchal stuff)

So what do you think everyone? Good idea or should we wait for a few more server compromises before we think about securing software repositories?

Finkployd

Re:So what do we do to prevent this in the future?

[Minna Kirai]

With this "Public Key Cryptography" you could conceivably sign software in such a way that it could not be altered without breaking the signature,

No... the way to alter software is easy to conceive.

You simply have to hack into the computer holding the private keys used for the signing (very likely the same computer holding the source code as well, and the system which normally uploads new packages to the distribution point). Once there, you can make changes and sign them just as if they were official.

Since attackers of this type have already demonstrated an ablity to hack into computers, PK signing doesn't add any true security. It adds some defensive obscurity, since it's more difficult for the attackers to locate a developer's machine than a distribution one. But dev systems will be more vulnerable to hacking, since they're not likely to apply patches as quickly as a public server. (And I won't recite the old line about "security through obscurity")

The only true benefit from PK signing is that end-users are protected from poor security at mirror sites. Suppose your ISP offers a Debian package mirror as a high-speed convenience, but doesn't secure it well. If it's compromised, trojan packages could be sent to you on the next "apt-get". Comparing those packages against signatures from an official debian.org site will protect you. But that assumes the official servers are sufficiently well-run to avoid being hacked. And as we've seen today, that's not the case.

Re:So what do we do to prevent this in the future?

[finkployd]

No... the way to alter software is easy to conceive.

You simply have to hack into the computer holding the private keys used for the signing (very likely the same computer holding the source code as well, and the system which normally uploads new packages to the distribution point). Once there, you can make changes and sign them just as if they were official.

Assuming you knew the password for the private key (private keys really should be encrypted with a password, especially for this).

Now before you go all 'keylogger' on me :) I will say that the private key should be kept on the personal machine of the person doing the signing, so they can dl the package, and sign it locally then upload it. Additional work? Sure but worth it in my opinion. What it really comes down to is that it is easier to keep a private key secure than it is to keep software that by definition is "open" for multiple people to work on secure. I mean if you cannot figure out how to keep a small private key secure, what hope do we have for free software's security?

Frankly, I'm more worried about mirror sites right now than anything else. Let's face it, there are tons of them, we do not know nor necessarily trust the people running them, and they are much less apt to reveal a compromise than someone lie GNU or Debian.

Finkployd

Where did you get those keys?

[dpilot]

Then the next point of failure becomes the keyservers. How do you know you imported a good key, and that the keyserver hadn't been compromised when you did it?

This probably would be no good as a way to sneak backdoors onto more than a few machines, since keys are usually stored once and used often. But it would be good to have some sort of key distribution and verification system. Imagine a key publisher having 7 peers, and where they carry same keys, requiring 5 to 7 matching signatures, and point a nasty finger at the odd one(s). More than two mismatching signatures and the system quits publishing keys.

Of course then the key publishers themselves then become a choke point for a DOS attack, of sorts. Make updates grind to a halt as a new exploit is emerging, widening the window to utilize it. But still, most keys are stored, and the voting fails only stop distribution and verification.

Thorny issues, part of why PKI is considered 'hard'. But at least my suggestion is reasonably decentralized (I didn't say how to get a new key into the system) and has publishers voting on the intersection of their published keys, not requiring every server to publish every key.

I Haven't Paid for Debian

[Bob9113]

This news made me realize how much I depend on Debian. At the moment, every one of my machines (four servers, three workstations, and a laptop) runs Debian. I've been running it as my primary OS for... two years? So far I haven't paid a dime for it. It is a nice advantage of Free Software to be able to use it for free, but given the fact that I'm way out of "try-before-you-buy" mode, I'm going to send them a check today. Software in the Public Interest was founded by and is the current funding source for Debian.

One server compromise in the two years that I've been watching by a company with zero product sales revenue is pretty impressive. An OS that is (IMO) dramatically superior to any commercial offering for free? They've earned my respect, and have clearly earned my cash.

Re:password

[jrexilius]

Actually there is a method for securing against lost passwords (by this I mean intercepted as in looked over shoulder, recorded key clicks, etc.) and that is the one-time password method combined with some other secondary authentication method. It is, however, extremely difficult to implement successfully. I have been kicking around a method creating my own system for this for my servers. I suspect that I wont be bright enough to do a good implementation of it though.

Of course this has nothing to do with the earlier post being both right and wrong. (right in the sense that Joe CTOs are dumber than a bag of doorknobs, and wrong in that it is not a technical reflection of relative security between MS and Linux).

GPG already!

[alexandre]

When are they going to force everyone to sign the package with GPG and have a warning like ssh when a key has changed when you dist-upgrade?

It's about time will all the server compromised these days...

SE Linux

[Tracy Reed]

Steve from Debian Security Audit project says this occurred due to a password goofup so this doesn't necessarily apply here but it easily could have:

Machine as important as these should be running some sort of Mandatory Access Control system like SE Linux. I have done an evaluation of all of the root exploits I could find over the last few years and SE Linux would have prevented every one of them because the MAC system prevents unauthorized priviledge escalations. You can test drive my SE Linux box by telnetting (not ssh) to selinux.copilotconsulting.com with user root and password root.

Re:What was that about Windows servers?

[noahm]

If passwords are at fault and sshd was the service that was comprimised then get rid of the passwords and use RSA challenge-response authentication.

Unfortunately, I believe that that's already the case, and has been for as long as I've been a Debian developer. I believe what really happened is that somebody's home account or something was compromised, and they did the stupid passwordless ssh key thing (instructions for which are even on the Debian devel web site!). Even if they didn't use passwordless keys, rootkits with tty-loggers make it pretty easy to sniff a key's password if it's typed over the network.

noah

Am I the only one?

[stonecypher]

You know, an enterprising attacker could just pull the trust network down. Someone with sufficient skill could very easily just work on Debian for five or six months, get trusted, and embed a subtle bug into a remote point.

I mean, we can't find the unintentional ones. What makes you think we could find one chosen for its obscurity?