Two FreeBSD Project Servers Hacked

hypnosec writes "The FreeBSD project has suffered a security breach. Hackers have successfully compromised servers that were part of the infrastructure used to build third-party software packages. The Security team over at the FreeBSD project is of the opinion that hackers were able to gain access to the servers using legitimate SSH keys and not by exploiting any operating system vulnerabilities. Instances of intrusion were first detected on November 11. FreeBSD project, through a message on public announcements mailing list said that the security breach hasn't affected the project's core components like kernel or system libraries but, has affected third-party software packages being distributed by the project."

Comments

Yes, I read /. on Saturday

[alphatel]

This was already submitted two days ago.
New article link merely references the material already posted by freebsd on Nov 17th.

Re:Yes, I read /. on Saturday

[Jeremiah Cornelius]

Dupe, dupe, dupe,
Dupe of URL
Dupe, dupe,
Dupe of URL
Yes, oh, I, I'm gonna link you
Nothing can stop me now
'Cause I'm the Dupe of URL...

Re:Yes, I read /. on Saturday

[kwerle]

Sigh. I was actually hoping for new information. Instead we're left with "/. editors can't scrub for dupes." Which we all knew already.

Re:Yes, I read /. on Saturday

[LurkerXXX]

Yes, but it was posted bt Timothy, who I and many others block all stories from (because he's an idiot that doesn't bother to look at the pages he links to, or in general give any other thought to posting), so at least it's new to some of us.

at least its 36 hours since the original posting

[Lawrence_Bird]

Posted by timothy on Saturday November 17, @09:22AM
from the happy-transparency dept.

Has anyone found out how they got the keys yet?

[Viol8]

They're not something you can guess. Someone with access to those systems either was careless with them, let someone else use their account and they were stolen or its an inside job and they're simply trying to make it look like it was external hackers.

Re:Has anyone found out how they got the keys yet?

[zeroryoko1974]

Probably tricked someone into giving them up. Your security chain is only as strong as the weakest link

Re:Has anyone found out how they got the keys yet?

[Idbar]

Probably someone left the keys in a bar in San Francisco. Isn't that the way it works these days?

Re:Has anyone found out how they got the keys yet?

[dkleinsc]

My guess:
1. Somebody who legitimately has the keys put them on a cell phone or laptop.
2. Somebody else pwns that device (because it's not running a super-secure OS), sees the keys.
3. The person with access doesn't know he's been hacked, or doesn't want to admit it, so the rest of the organization doesn't get notified and can't change the keys.
4. Voila, easy access to FreeBSD's servers.

That's one of the standard techniques in getting around security: You target the relatively insecure partner with legitimate access to your real target.

Re:Has anyone found out how they got the keys yet?

[Anonymous Coward]

It concerns me that so many people (lots of people on forums, Slashdot, and a few of my own peers) are focused on how the person's SSH private keys were obtained. It doesn't matter how the keys were obtained -- truly it doesn't. You have to assume those keys are going to be obtainable. Most people keep their private keys stored on their workstation or laptop, or on a USB flash drive; laptops get stolen, USB flash drives get stolen or lost, workstations get compromised, and so on. Given this, there's no real way to guarantee someone won't get your SSH private key, and this is exactly why a private key should be password-protected. So unless a keypress logger was installed on the person's machine whose keys were stolen, I have to assume the SSH key in question was passwordless, in which case the fault here is with 1) sshd being set up to allow passwordless keys, and 2) the user using a passwordless key to begin with. (Also, for those considering a rebuttal along the lines of "passwords are annoying with SSH keys, you have to enter them every time" -- look into this amazing thing called an SSH agent).

The things that concern me most about this breach:

1. What I mentioned above,

2. Why two package-build cluster machines (which may or may not allow non-root users to write to the official package repository -- see #3) are connected directly to the Internet without (presumably) any firewall rules permitting SSH access from specific IP addresses (I can guess this is because most of the FreeBSD devs tend to roam internationally, but that's no excuse when it comes to security -- there are ways to solve this, both technical and social),

3. Why the account's access level was not stated in the disclosure. Did he/she have root or not? It matters -- if they didn't have root, why would there be any concern over the integrity of the packages on the official package FTP server? Unless, of course, somehow that non-root-level account had the ability to remove/overwrite/replace those packages on the official package server directly, which to me indicates a much bigger problem than the breach itself,

4. Why the FreeBSD Project members said nothing officially or in an official capacity until November 17th. Someone (I'm assuming Peter Wemm) made the decision to down most (but not all) of the related services on the 11th. Someone on the PR side (in this case, probably FreeBSD Core, or possibly Ken Smith) should have sent an Email to -announce or -stable or -questions, posted something on the freebsd.org website, RSS feed, or even Twitter, as well as the official FreeBSD forums, stating something to the effect of "we've shut off the SVN-to-CVS gateway for an investigative matter; you won't see ports/src/etc. updates until we turn it back on. We've also shut off the portsnap master. csup/cvsup, portsnap, freebsd-update, native CVS, and GNATS are all impacted. We'll provide more detail when we have it, rest assured".

I appreciate public disclosure of the breach (even though it leaves many questions), but WRT #4, a 5-day outage without any official announcement is completely unacceptable. There are already FreeBSD users on the mailing lists complaining about this fact, and so far not a single Project member has responded in official capacity to those concerns.

I'm also choosing to exclude a 5th item from my list, which is why 2 or 3 key FreeBSD Project members tried to sweep this under the rug by stating on mailing lists it was "maintenance". That indicates people were either in the know but were mum about it, or didn't know and were spouting off speculative nonsense (both of which are unacceptable in this situation).

What Timothy's news item, as well as yesterday's news item, lacked was what FreeBSD services were impacted and for how long. I submit a Slashdot story (which never made it past the firehose approval process, not sure why) which disclosed what all services were impacted:

Re:Has anyone found out how they got the keys yet?

[icebike]

Look hard enough and you will find a conspiracy.

It seems just as likely to me that forensics required a certain period of silence while packages were checked against backup sources.

What could they announce on the 11th that would have made you happy? WE'VE BEEN BREACHED!! (perhaps add two or three more exclamation marks). Then what. 10 thousand questions, phone calls, and emails, distracting them from the task at hand?
You know that even you would be demanding more answers if they posted exactly what you asked for in #4.

The 5 day outage was TOTALLY acceptable and appropriate. If your systems can't run for 5 days without contact with the mother ship, then you need a different system. The Silence was acceptable too.

Mountain / Molehill.

Re:Has anyone found out how they got the keys yet?

[icebike]

Well even having found a cell phone with ssh keys on it doesn't gain you access unless the ssh keys themselves have no passphrase.

This use to be a fairly common practice (unfortunately) when key caching agents were not available and every single transfer over ssh required yet another entry of your ssh passphrase.

If no passphrase was used on the keys, simply walking away from your workstation for two minutes allows an untrustworth co-worker to email your entire .ssh directory to himself at some obscure mail site, or copy them to usb.

By far the likeliest situation is a ssh key with no passphrase.

Should have run on OpenBSD

[HaeMaker]

"Only two remote holes in the default install, in a heck of a long time!"

Re:Should have run on OpenBSD

[Zemplar]

"Only two remote holes in the default install, in a heck of a long time!"

A security breech using legitimate authentication credentials is not a remote hole.

Still suspect this could have something to do with

[RocketRabbit]

Still suspect this could have something to do with the SSL backdoor allegations made a while back. http://www.mail-archive.com/full-disclosure@lists.grok.org.uk/msg47029.html

Yes I know the allegations have largely just petered out over time, but this doesn't allay my suspicion.

OpenBSD would have been worse

[Onymous Coward]

Indeed.

And this is why there's an essay on how OpenBSD is insecure. Because security breaches do happen. You need to lock your systems down against internal attackers as well. Defense in depth and all that. Not just hermetically sealing your system by abstaining from running any services at all.

Re:"Passphrases"

[icebike]

Its as easy as simply running a dictionary attack.
You can't tell a pasphrase protected private key from an unprotected one. Both are gibberish. You would never know when you
decoded it correctly unless you try to use it.

Each dictionary attack attempt will have to be tried via an attempted log in to either the target site or a replicate there of.

But, hey, we are all ears if you have a better method. People have only been looking for one for something like 20 years. You can be a hero.

Re:"Passphrases"

[fnj]

..are a shitty concept, as attackers have enormous compute resources these days, combined with sophisticated dictionary attack software. $100 from a creditcard can buy you serious compute power on AWS.

Perhaps you're not very good at statistical analysis, or you just haven't gotten the message. Yeah, they're not a magic bullet. You have to pick a decent one. But you can remember a HELL OF A LOT more entropy in the form of a phrase than you can in the form of a nonsense character string.

Of course you don't KNOW they are using a "good" passphrase. You don't know they haven't written down a password, either. Like icebike says, maybe you have a magic solution that works with no brain or discipline required. Let's hear it, genius. Yeah, I guess I just insulted an anonymous idiot.

Damn pirates

[bursch-X]

And the worst: They stole all the source code and pirated BSD!!!!

"hacked"

[smash]

It's not really a hack if you log in with legitimate credentials. Compromised, yes. Hacked? No.

Re:"Passphrases"

[smash]

A passphrase is better than NO passphrase.

Re:Malice vs Incompetence

[RocketRabbit]

Linux, Windows, OS X, and Solaris all use the BSD SSL code, or very close derivations of it. If the BSD coders are lazy, then the coders responsible for the above-mentioned OSs are even worse, right?