Debian Project Servers Compromised
Sean was one of many to pass along
the bad news
from the debian-announce mailing list: "Some Debian Project machines have been compromised. This is a very unfortunate incident to report about. Some Debian servers were found to have been compromised in the last 24 hours. The archive is not affected by this compromise! In particular the following machines have been affected: 'master' (Bug Tracking System), 'murphy' (mailing lists), 'gluck' (web, cvs), 'klecker' (security, non-us, web search, www-master). Some of these services are currently not available as the machines undergo close inspection. Some services have been moved to other machines (www.debian.org for example). The security archive will be verified from trusted sources before it
will become available again." They were going to announce 3.0r2 this morning; they've checked it and it's unaffected but obviously they're still postponing that release.
The debian-announce archive [ http://lists.debian.org/debian-announce/debian-ann ounce-2003/threads.html ] doesn't list this message. Of course with the number of machines affected it's possible that the mailing list archive is somehow affected.
-JohnF
Why my apt-get was failing from people.debian.org last nite. Not to mention why debian.org was down. :(
MD5 sums are used for the contents of packages, but packages may only be uploaded and processed by the build system if they're correctly signed.
So yes it's not trivial to backdoor a package - unless you're already a Debian Developer...
Which is why using something similar to ajt's apt-check-sigs. (google cache, since people.d.o is down.)
Indeed, that's one of the few areas where the Debian Project has lagged behind other distribution vendors technically - cryptographic signature verification for packages.
This infrastructure has been kind of long in coming, but as of a few months ago, you can now verify Debian package signatures with debsig-verify. Might I suggest everyone install and use that?
"How can you claim that you are anti-crack, while still writing a window manager?" — Metacity README
Security is much much more than "just keeping your system up-to-date".
...) to log in to one of the servers
- accounts can be compromised
- unknown bugs may have been exploited (although that's unlikely in this particular case)
- crackers could have been cracking a developer's system, and using information they find on that developer's hard disk (ssh key, gpg key,
- also of importance in general is the competence of the administrators (which surely is *not* at the cause of the problem here).
Of course these systems are running debian stable; but that's most likely not the problem.
Yes Debian's machines run Debian, this breakin wasn't anything to do with the software installed upon the box, as it was due to a password compromise.
If anything it's more embaressing that somebody lost their password than that the software wasn't up to date.
here.
r ity-20031121.txt | gpg --verify
To verify it:
$ wget -O- http://cert.uni-stuttgart.de/files/fw/debian-secu
(drop the space, of course)
Assuming you trust the key it was signed with, of course...
At least cjwatson and myself are Debian developers. I wish I could say it's a hoax, but it's not. However, as you've already read: the archive doesn't seem to be compromised at all.
the distribution contain a Release.gpg file that is signed: so it is not possible, for example, to compromise a mirror, and it is more difficult for an intruder to compromise a single Debian package in the archive.
There is a script apt-check-sigs that will check the above signature: this is explained in the debian page on releases; unfortunately the link to download the script from there is down, here are two alternatives: google cache my site (slow)
yep, GPG signed... the public keys of all the developers are avalible on http://keyring.debian.org normaly, and it still appears to be up anyway. There is also a debian package which contains all the keys too
--- snip here ---
K ik pLMtJKcxSKUgvyn NTi4sT01MVEtMTM/ OKS4CCqQrZqZUK1 K5dHW5OuyZWUE27o M5QZDp9w6GBQtO+ hFp+fRBXM7HXcYc1 6Xj5A9DwA=
This is a truthful report.
You may validate this message against the key for skx@debian.org.
Steve
--
-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.2.3 (GNU/Linux)
owGbwMvMwCR44PyxzWd9eOcyns5PYrDfJ7EiJCOzWAGIEhV
i0r0uLgi80sVchMrFcoSczJTEktSFUpAi
aflFCsXZFQ4pqUmZiXl6+UXpQCO4gktSy
SLxI+1madnvjbIZVrZu0HcTnzGdY0LBFy
=xVtr
-----END PGP MESSAGE-----
I don't know if this delayed a release, but -- in October 2000, the news broke that Microsoft's internal network had been cracked for three months.
(Debian made this announcement in 24 hours.)
Read for yourself:
Microsoft Cracked
If you care about security, you're only using Debian stable.
;-).
;-)
If you're using stable, the only updates are security fixes and point releases. Both are annouced by signed emails before hitting the archive.
So just don't blindly update & upgrade on a whim. Instead, regularly check the annouce/security lists, and only upgrade when required. For the common case (security update), you'll also know exactly which packages apt should flag for update (you can also do the exercize for a point release, but it's more work
Of course, that's not 100% bullet-proof. The archive could be compromised so that just a security updated package is "trojaned" for example. But that's harder.
Practical exampl: this morning apt got errors and suggested upgrades never annouced for after an update. I smelled something bad and did not upgrade, waiting for just this kind of news
And I haven't been disappointed: the reaction has been quick & honest, and no harm on my side...
Just in the interest of full details, BitKeeper was NOT compromised. The CVS bridge to BitKeeper was the software that was compromised. BitKeeper caught the problem and did not let the back door into the kernel source tree.
-- http://www.MindBlowingPhotos.com
Photography inspired by music, nature and life itself.
PGP keyservers (unlike, say, Kerberos KDCs) are completely untrusted. Anyone can upload any key to a keyserver. And downloading a key from a keyserver implies nothing about that key.
To verify that you have a valid key, you have to rely on the web of trust. Basically, if a key is signed by someone whose key is signed by someone [recurse through however many levels you are comfortable with] whose key you have personally inspected, then the key can be assigned a trust metric based on how reliable you consider that chain of signatures to be. (Basically, how much you trust the integrity and acuity of the people controlling the chain of signatures.)
PGP and GnuPG have supported this infrastructure from Day 1. Asking people to trust an arbitrary third-party public keyserver was never in the plans.
"How can you claim that you are anti-crack, while still writing a window manager?" — Metacity README
It isn't hard to create a key, upload it to the keyservers, and sign your backdoored glibc.
So unless you can trust the entity who signed the package, it's all moot.
Obviously, the debian project could sign the package using the Debian Package Signing Key, but you've just changed the problem from "how can an end user know that this key is worth trusting" to "how can debian know that this key is worth trusting". This is (probably) solvable, but still quite hard.
Note that the technology is easy, but the processess to back it up aren't.
I have no idea what the debian people are changing in their packaging system, but as long as new debian cds come with one public key that signs the rest of the keys on that server, and the corresponding private key is kept offline, there is nothing an attacker could do if they compromised the server. Any new keys they add would not be signed by the key that every debian user already has.
Precisely as hard as it would be on any other system, excluding those Debian boxes which actually verify the signatures before installing packages (where it would be impossible).
However, it would be noticed rapidly and suitable announcements made.
I hate to say it, but Microsoft's haven't been compromised, and they're the bigger target.
Not true.
Everyone here knows if windowsupdate.microsoft.com had been compromised, people would be droning on about how it's some sort of illustration of Microsoft's security.
Their update server wasn't compromised, but the debian archive also wasn't compromised in this case. But, yes, we have to work harder to make our servers secure. And we will never reach the point were our systems will be unvulnerable. So what is your point? You complain that there aren't enough anti-oss-trolls here?
Martin Schulze is also in the Debian security team. He prepares a lot (most?) of the security fixes for stable.
I'd hate to say this too, since it is wrong.
Microsoft's internal network was compromised, as reported by the BBC, and many other news agencies.
So, please do some research before welcoming your "secure" overlords...
As far as I understand, no machines apart from the several Debian computers have been compromised. Compromising a machine that hosts the central Debian APT repositories is a perfect opportunity for backdooring thousands of machines In this case, that didn't happen. "Thousands of machines across the globe" have not been compromised. I guess it's only good luck but Debian users were not affected by this security breach.
"Where are we going, and why am I in this handbasket?"
I don't think it was foolish. If you used the ISO images to do installs/upgrades, the GPG keys were obtained from there. And with [signed] md5 checksums available to verify the images, you know the GPG keys that RPM uses to verify the packages is trustworthy. Since you can have faith in the keys, you can have faith in the package. this has, in fact, long been one of the things people traditionally point out in the deb vs rpm holy war, in favor of rpm. From the comments I'm seeing, it looks like GPG checking is being added to apt in debian (the apt pacakges on freshrpms.net and fedora.us for Red Hat and Fedora Core already use rpm --checksig). I think it should be added to dpkg, so then apt can just relegate the verification to the actual package installation tool.
Read up on Debian before asking such a rhetorical question. You become a debian "coder" (read: package manager) by finding a package or two to work on, and working on them. You don't directly affect Debian, and you're basically a lackey.
Skip ahead a year or two. If you're doing a good job (and I mean a _damn_ good job), someone might propose you become a Debian Developer. Here is where the danger lies, and I'm not sure how a DD let their password slip.
But certainly don't think that because you have some 2-year college course under your belt that you are going to jump into the think of Debian development right away.
did you read ANY of the posts before trying to sound like a genius revolutionary ??
apt-secure
apt-check-sigs
not to mention they are already gpg signed
Redhat, +1, Already doing it. -1, not failing to install if the packages don't verify.
Which is exactly the state in Debian, too.
Jeremy
Looking for a Python IRC bot?
Their update server wasn't compromised
It has been before. when code red hit. Although the link given in that article is no longer working there are plenty of screen shots of www.windowsupdate.com with 'hacked by chinese' on it out there somewhere.
You cannot blindly trust anything, from anyone. I don't care if Mom says her apple pie is just dandy I'm gonna run my own tests.
you're all figments of my deranged imagination
Finally, the NM process itself is the ultimate arbitrator of who enters Debian. A prospective developer gets evaluated by multiple people before he or she actually becomes a developer.
While still not foolproof, these techniques combined help reduce the lack of accountability and the lack of trust in the system. [Of course, in the end, you really need to go out and sign and get your key signed by a Debian Developer (or a couple) so you can join the web of trust and the strongly connected set too.]
http://www.donarmstrong.com
What "lots"? The worst thing that can happen is getting everybody's key revoked. Sure it could be quite a mess, but other than annoying everybody and forcing the developers to make new keys it wouldn't really accomplish anything.
A revokation key has little attractiveness, IMO. By most part, having your key rekoved doesn't stop you from communicating, nor it allows whoever got it pretend it's you. Nothing stops you from having more than one key either. You don't have to use the Debian one for everything.
Safekeeping is easy too. Print it on paper (it looks same as a PGP ASCII-armored key), store somewhere safe (put it in a bank for safekeeping) and then agree that when there's enough people who think the key should be revoked, go fetch the paper and type the key on the computer.
There's really no reason to keep them on a computer. Revoking your key isn't something you do often.