Linux Foundation, Linux.com Sites Down To Fix Security Breach

An anonymous reader writes "All Linux Foundation sites seem to be down due to a security breach, which occured on 8 sep. (according to a notice displayed on the site)." From the email I received this morning, sent to all Linux.com and LinuxFoundation.org users: "On September 8, 2011, we discovered a security breach that may have compromised your username, password, email address and other information you have given to us. We believe this breach was connected to the intrusion on kernel.org. As with any intrusion and as a matter of caution, you should consider the passwords and SSH keys that you have used on these sites compromised. ... We have taken all Linux Foundation servers offline to do complete re-installs. Linux Foundation services will be put back up as they become available. We are working around the clock to expedite this process and are working with authorities in the United States and in Europe to assist with the investigation."

Comments

SSH keys?

[betterunixthanunix]

Uh...isn't the point of using public keys that you do not have to keep them secret to remain secure? If people uploaded their public keys to the compromised systems...how is that a problem?

Re:SSH keys?

[smash]

Ahh, but if you were a dumb ass and placed your private key for linux.org on say, kernel.org (to log into one from the other), then if kernel.org got hacked, your key is gone.

Re:SSH keys?

[maswan]

It is an unfortunately common case that people copy/create private ssh keys on servers to login (or scp) from those to another remote host. These keys are of course compromised.

Re:SSH keys?

[kestasjk]

Why would you do that?.. Usually people sacrifice security for convenience, but that is less convenient..

Re:SSH keys?

[Anonymous Coward]

Dumbass is right. That's why OpenSSH has agents (and the ssh agent does NOT expose your key, it does all crypto and returns just the result). SSH private keys should NEVER leave your own box. You don't reuse openssh keys, either. It is one private key pair per host.

That would not have saved HPA's keys since his laptop was the entrypoint of the kernel.org breach, but if some people were not lazy assholes, there wouldn't be any private keys on kernel.org, except for some automated tasks (which have to be immediately revoked).

Also, you NEVER "hop-jump" SSH, even when using an agent: you do port forwarding through that host instead. If you do "ssh hop-jumping", a compromised system can record anything you type inside that ssh session, such as root passwords. This is exactly why the openssh server and clients were trojaned in kernel.org (and probably linux*.com).

Re:SSH keys?

[dbrian1]

I don't see why having private keys on a server would be less secure than having these on your laptop/phone, which is much easier to steal or borrow...

My laptop is only vulnerable to theft by people I am in physical contact with and is generally my responsibly to secure while connected to the Internet. Placing SSH keys on a server means I'm giving these keys and any access they grant to the admins of said server and am placing my trust in them to keep them secure. This is fine for automated trust relationships between hosts but not generally a good idea for personal keys.

Re:SSH keys?

[maswan]

Well, you'd still need keys on your laptop to get to the server. So now you have two places where your keys can be stolen and used to login everywhere you trust your keys.

For the case where you actually do need direct communication between two servers you probably want to do agent forwarding instead of having more keys in your authorized_keys. Remember that every single entry there is a point of failure, and any one of them getting compromised means that your account is likely to get owned.

Now there are special cases where having more keys is useful, but most of the time they just open up more vectors for someone to steal them and break into other computers.

Of course, even then, they are better than passwords, at least if they have proper passphrases. Not too uncommon to see lots of passphrase-less keys in home directories on multi-user servers though.

Re:SSH keys?

[smash]

Because you're a dumbass.

Re:SSH keys?

[buchner.johannes]

It is an unfortunately common case that people copy/create private ssh keys on servers to login (or scp) from those to another remote host. These keys are of course compromised.

There is no requirement to do that. You can just create a tunnel through the first server to the second (ssh -L). Then you connect to the tunnel port on localhost, and you never had to give away your private key. You're even safe if the server in the middle gets compromised.

Re:SSH keys?

[ArsenneLupin]

Why would you do that?

Maybe in case you'd need to scp large files from linux.org to kernel.org and vice-versa, without having to first download them to your home machine, and then upload them to the target (slow, if you've got asymetric DSL)

Of course, dumb users not aware of security implications is also a possible explanation (less likely though, as dumb users would probably not be aware of ssh authorized_keys in the first place...).

In both cases, it is smart of the Linux Foundation to warn their users of this potential issue...

Re:SSH keys?

[ArsenneLupin]

That being said, it's important to use a different private key on each machine where you might ssh from...

However, in case of a compromise, you'd still need to remove trust in the private key of the impacted machine (so if kernel.org got hacked, you need to remove your old kernel.org's public key from your ~/.ssh/authorized_keys on linux.org)

Also, if the hacker got kernel.org's /etc/ssh/ssh_host_rsa_key, then he could theoretically later on mount an MITM attack against kernel.org... so the kernel.org admins better change that one as well (while publishing the new key's fingerprint on an SSL server).

Re:SSH keys?

[doublebackslash]

Why shouldn't I re-use my ssh keys? I'm pretty clear on all your other points so I assume that you've got a good reason for this. What is it?

I feel that I may have mis-understood the proper use of SSH keys and would really like to know the attack vector here.

Re:SSH keys?

[rastos1]

Maybe in case you'd need to scp large files from linux.org to kernel.org and vice-versa, without having to first download them to your home machine, and then upload them to the target (slow, if you've got asymetric DSL)

That's why there is -A option for ssh.

Re:SSH keys?

[ttong]

I once had a co-worker who was as stupid to do that, so I immediately blacklisted his key on every machine on the corporate network and told him to generate a new one and keep the private part to himself. I sincerely hope any sane IT dept. will do the same in that event.

It's almost as stupid as emailing the root password in the clear for the new dedicated server you've ordered. Mail me the host key fingerprint, I'll mail you my public key. How hard can it be?

Re:SSH keys?

[smash]

Also - don't forget, if someone has your public key (and has compromised the server's key), they can impersonate the remote end and go for a man in the middle attack.

Re:SSH keys?

[doublebackslash]

The statement that I was in disagreement with was "You don't reuse openssh keys, either. It is one private key pair per host."

Bullshit unless there is something that I'm missing. Keep one private key / public key pair and use the public key on all things you want to access. Keep the private key private (read: in a private place, encrypted. As is traditional.)

Is there some problem with re-using your public key on several servers? I think not.

Re:SSH keys?

[WindShadow]

That being said, it's important to use a different private key on each machine where you might ssh from...

Hell yes, and for key management sanity it is good to use a clear comment on keys so you know what they are for.

However, in case of a compromise, you'd still need to remove trust in the private key of the impacted machine (so if kernel.org got hacked, you need to remove your old kernel.org's public key from your ~/.ssh/authorized_keys on linux.org)

You also have to rebuild authorized_keys to check that no new or modified keys have been added. That would allow the hackers right back in at some future time. I keep a copy and cksum of authorized keys for every machine where I use keys, just so I can check. Yes, I'm paranoid.

Also, if the hacker got kernel.org's /etc/ssh/ssh_host_rsa_key, then he could theoretically later on mount an MITM attack against kernel.org... so the kernel.org admins better change that one as well (while publishing the new key's fingerprint on an SSL server).

I don't know if that's the case, but they could certainly set up a fake server if they have your public key and the server host keys. So generating a new host key pair is really required, forcing every user to change the key.

What a mess!

Re:SSH keys?

[ttong]

Yes, you are correct. Nothing wrong with the same public key on 100+ servers.

Though I can also relate to having a separate key pair for internal servers and another for customer servers. That way, when you use an agent and unlock a key you don't unlock every server you ever have to manage anywhere. Especially when you have a multitude (say, 5 at max) of key pairs for that purpose.

More Info, and Announcement Content

[LinuxScribe]

A few more details of the breach, including the content of the message from the Linux Foundation, can be found on ITWorld.

LinuxScribe

Re:More Info, and Announcement Content

[WindShadow]

The information I really want to see is a statement clarifying this as either technical as in a failure of the security software somewhere, or administrative, as in someone left something open through error or poor security design choices.

It's important to know if this is a bug which is on all Linux systems, or someone made a human error.

Lucky

[Anonymous Coward]

The attack that compromised some high-value servers belonging to kernel.org — but not the Linux kernel source code — may have been the work of hackers who simply got lucky and didn't realize the value of the servers that they had gotten their hands on.

Sure.

Connection?

[CheShACat]

First Kernel.org and now this? Has someone got it in for FLOSS at the moment?

Re:Connection?

[Nerdfest]

No, someone would like unrestricted, undetected access to make 'modifications' to the most popular server OS.

Re:Connection?

[doublebackslash]

Those are just pubic facing servers, not the dev team's workflow boxes. However, even if they were their source code management tool, git, exposes every change made by anyone for all time. It is simply that nature of git. Check it out if you want details. Even if they didn't use git the maintainers and anyone with clean copies (per-breach) can simply run a diff on the source code vs any new "untrusted" copy they may see.

Since they've made the breach public and the aforementioned steps to detect any modifications are trivial for anyone to perform (Linux is not mirrored on github, a public git repository and absolutely wonderful site) there is a very low risk to anyone and zero risk now.

I'm not going to stand on a soap-box here but I'll just say this:any decent company or organization that is the victim of an attack such as this should have procedures and systems in place that render it as harmless as this one. Think of the impact of this hack compared to others in the recent past and near future. It might provide a wonderful 'dos and don'ts' guide

Total Irony

[adosch]

I am not sure what is gained by breaking into, compromising and exploiting a foundation like The Linux Foundation when the perps were more than likely using a Linux driven OS and tools to do what they did. Seems like some need a re-education in not harming the hand that builds your house...

Re:Total Irony

[phoric]

Follow the money. Who stands to gain from creating fear, uncertainty, and doubt surrounding Linux and it's related organizations?

Re:Total Irony

[Anonymous Coward]

Follow the money. Who stands to gain from creating fear, uncertainty, and doubt surrounding Linux and it's related organizations?

Exactly! The folks at BeOS!!!!

Re:Total Irony

[PNutts]

Follow the money. Who stands to gain from creating fear, uncertainty, and doubt surrounding Linux and it's related organizations?

Exactly! The folks at BeOS!!!!

BeOtcheS. BeOtcheS love FUD.

Re:Pro-hacker when I want

[Stormthirst]

Mod -1 Troll

Seriously?

Re:linux security is a joke

[Stormthirst]

Mod -1 Troll

Learn to use the '. Oh no - you're right. You are Ill.

Re:Pro-hacker when I want

[Elbereth]

Do you ever post anything other than instructions on how to mod other posts?

This wouldn't have happened with OpenBSD!

[Anonymous Coward]

Having said that, reasonable people may conclude that the occasional security breach is an acceptable price to pay to avoid dealing with Theo. :-)

This is the right way to do it

[Pop69]

Not like when a CA gets its webserver compromised, has a quick self audit and then declares everything is OK, really, honest....

Assume everything is compromised unless you can prove otherwise and get the staff in on overtime to reinstall from scratch.

Re:linux security is a joke

[hcs_$reboot]

Breaches: Linux 1, Windows 2317
Still some margin...

Re:Inb4 OS Slapfight

[HiThere]

I don't see any reason to exonerate either of those, but you need to widen your focus. You should include numerous governments (or their agents, perhaps acting without authorization). You should also include groups of criminals, like the Russian Mafia. Then there's folks just out for lulz. etc.

Some of these may be more likely than others, but don't narrow your focus too much. Not without reasonable evidence.

Linux fail security?

[Ryanrule]

Thats unpossible.

SSH Keys Compromised?

[swillden]

How could an attacker getting hold of the public key "compromise" anything? It doesn't contain any personal information, and -- barring an earth-shattering breakthrough in cryptanalysis of RSA (or DSA, if you chose a DSA key pair) -- it can't be used to gain access to anything, not even the system it was stolen from.

That's the whole point of using asymmetric cryptography for authentication!

Re:SSH Keys Compromised?

[smash]

You're assuming only public keys were compromised. If they had private keys enabling login to other hosts stored on said systems (yes, retarded, but...) then those keys are compromised.

Re:SSH Keys Compromised?

[swillden]

You're assuming only public keys were compromised. If they had private keys enabling login to other hosts stored on said systems (yes, retarded, but...) then those keys are compromised.

A little more than retarded... in order for someone to do that they'd have to have never heard of ssh-agent.

Re:SSH Keys Compromised?

[Antique Geekmeister]

ssh-agent leaves an active and unlocked ssh-key available on the relevant server. Many of my colleagues refuse to run their ssh-agent as part of their own persoanl host's working environment, and prefer to host them on their most used server, despite my warnings. Others find ssh-agent burdensome and simply use passphrase-free keys, and there is not yet any graceful way to prevent that except to audit for them on the client hosts, which is awkward and intrusive.

Re:ssh keys compromised?

[Alex Belits]

THEIR host keys and everything that can be decrypted using those keys.

You don't get it.

[Anonymous Coward]

You don't get it. The difference is Microsoft is a giant multi-billion dollar corporate machine with an insane marketting and advertising arm that helps them get into places that Linux simply can't because of lack of finances. If a well-funded corporation slips up its schadenfreude because 'even with all that' they still couldn't get it right. While Linux enjoys SOME corporate backing, its still largely a labour of love by independant developers. Most people side with the little guy. ;-)

 

Re:You don't get it.

[cavreader]

Linux enjoys more than SOME corporate backing. Look at the shear number of web servers and all of the Linux derived operating systems currently being used in the mobile device market and you will see quite a few big corporations supporting the platform even if they don't distribute their applications or make their changes available to the OS community. Your little guy labor of love argument disappeared about 10 years ago.

Re:You don't get it.

[kiwimate]

I think you're the one who doesn't get it. Linux is ubiquitous, and has been for several years, in the enterprise server world. Granted, the post you responded to was a bit of a troll, but you've missed an important point.

You can bark all you like about the marketing strength of Microsoft, but the CSO doesn't much care about marketing if the FBI comes visiting because you just self reported a hack and your company is an enormous financial institution, has millions of health records on file, or is part of the critical infrastructure.

Whatever the reality of who is behind Linux, the end user isn't going to be swayed by an argument that boils down to "people shouldn't hack us because, wow, golly gee, Linux is the show we can put on in our own back yard, mister". And all the arguments about costs aren't that strong; the licensing costs are obviously in favor of Linux, but the support costs are usually about the same between deploying Microsoft and deploying Red Hat.

I know it's conventional on Slashdot to run down the stereotypical technically illiterate bean counter with an MBA, but my experience has been that most companies have fairly hard nosed and at least somewhat technically astute people in positions of responsibility. They may not understand the ins and outs of SQL injection, but they are going to be less than receptive to the argument that "people won't hack us on Linux; it'd be rude".

Re:Inb4 OS Slapfight

[smash]

OP is not a troll. This and other 'hacks' are obvious attempts to destroy the infrastructure behind Linux.

I wouldn't go that far. Sounds more like a few attacks on high profile public systems that just happen to host linux projects.

Re:linux security is a joke

[smash]

No OS is secure. Take precautions. Having random members of the public from around the world with abilities to get shell access on your systems makes it hard.

Re:corporate espionage?

[smash]

...And, here's your hat.

Microsoft doesn't need to resort to this sort of thing to demonstrably kick linux's arse in the market place (i've been waiting for year of the linux / unix desktop since 1995).

More likely, its some random spammer / organised crime group looking for either a well connected site to launch DDOS from, or to insert malware into the linux kernel for spambot purposes.

Or a script kiddie doing it for bragging rights.

The Joker In The Deck

[westlake]

I haven't seen anyone talking about the elephant in the room: Just who would stand to profit from manufacturing FUD surrounding Linux as a result of security breaches?

Alfred Pennyworth: A long time ago, I was in Burma...working for the local government. ... One day I saw a child playing with a ruby the size of a tangerine. The bandit had been throwing them away.

Bruce Wayne: Then why steal them?

Alfred Pennyworth: Because he thought it was good sport. Because some men aren't looking for anything logical, like money. They can't be bought, bullied, reasoned or negotiated with. Some men just want to watch the world burn.

Re:Obvious Troll...

[Slashdot Assistant]

Surely not! What gave me away?

ssh keys compromised?

[madbavarian]

"you should consider the passwords and SSH keys that you have used on these sites compromised."

How the heck can ssh keys compromised by this breakin? Doesn't the site just have access to the developer's public key? With a sufficiently large ssh key (say 1k or 2k) how is anyone going to derive the ssh private key from the public key? The fact that if is effectively impossible is supposed to be the whole point of public key encryption.

Re:linux security is a joke

[PNutts]

Breaches: Linux 1, Windows 2317

That's the problem. Complacency? Ignorance? Denial? Or just another bash on Windows?

Re:Fedora just pushed a kernel update..

[inode_buddha]

Try "rpm -qil "

Re:hahahahah

[aztracker1]

They don't post every Windows security bug. There tends to be a posts on automated exploits/worms/virii in the wild and major encroachments such as this. MS has a tendency, for better or worse, to patch security exploits in their regular release cycles, unless compromised by an automated exploit. MS has made some very poor security choices over the years in favor of their development cycles, and release schedules. Don't get me wrong, I happen to like a lot of things that have come out of MS, but that doesn't absolve them of anything.

Today, just about the most unforgivable sin is not using parameterized queries with database access. It still happens though, and there's lots of cruft code out here written by those who didn't know any better. I would suspect there's more than a handful of areas in systems that weren't truly given a thorough code security audit. It happens, the best you can do is fix what you find, and handle known exploits in the most transparent, and expedient way possible.

Re:linux security is a joke

[hcs_$reboot]

That's the problem. Complacency? Ignorance? Denial? Or just another bash on Windows?

Bash on Linux, actually.

Re:Inb4 OS Slapfight

[priceslasher]

You might also wonder if it is the new guy being incompetent or if someone has defected from within.