Slashdot Mirror
Gentoo rsync Server Compromised [updated]
costela writes "LWN points out that the Gentoo project
fired out an alert about one compromised rsync server." From the message itself: "However, the compromised system had both an IDS and a file integrity checker installed and we have a very detailed forensic trail of what happened once the box was breached, so we are reasonably confident that the portage tree stored on that box was unaffected." Update: 12/03 22:54 GMT by T : One more damage report: gibson writes "The Free Software Foundation recently discovered that its software host site was compromised a month ago. The compromise appears to be the same as the recent attacks on the Debian servers. The site is shut down until Friday while they install replacement hardware and verify the authenticity of the hosted source code."
who didn't see this coming? I use gentoo and i figured it was a matter of time before someone did this. I mean haveing a central tree is cool but it does make it more of a target for attacks. I am however glad to see that they took precautions.
A conspiracy theorist could have a field day..
Now where did I put my tin-foil hat?
Is it even a linux box? Just because it's part of the gentoo rsync respository network doesn't mean it's running gentoo, or indeed linux.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Either hackers have decided they *hate* OSS (not likely) or someone is putting up a purse trying to damage the OSS communities security image.
Quack, quack.
remote logging
I just threw away my tinfoil hat and made a new one out of steel. With a spike on top.
Once is happenstance, twice is coincidence, three times is some one playing silly buggers.
(Kernel.org, debian.org, gentoo.org - all in the same two months?)
Beep beep.
They haven't had a break in two weeks!
Any bets on which major distro will be next? Better yet, instead of point spreads on professional sporting events - Vegas should be taking bets on which distro (or well established free software org) gets rooted next...
First Debian, now Gentoo... Slackware perhaps? Maybe install a spam-bot on a knoppix image?
The kernel exploit was a local one. Evidence has shown that the exploit used against the rsync server was remotely exploited. Good thing that if anything had happened (which nothing has shown up yet, according to the #gentoo-dev channel on FreeNode), it would have only affected about 20 users. Still sucks that there's an exploit at all though.
"You tried your best and failed miserably. The lesson is...never try. Heh!" -Homer
"How come we never hear about breakins [at windowsupdate.microsoft.com]..."
Because we wouldn't have time for all of the other news.
Do not look into laser with remaining eye.
... they DO have records of what was done and were able to isolate it pretty quickly. IMHO, that's probably saved them a lot of trouble.
Whether it's because the cracker was sloppy or inexperienced, or because the Gentoo team have good server security, I can't say - but it seems they were pretty lucky compared to Debian.
What baffles me is why crackers go after targets like this. I can understand anticapitalist stuff, but my intuition says someone trying to crack a *nix server and damage a distro must have detailed knowledge of *nix systems - and is therefore likely a user of an OpenSource operating system.
Is that guess a little too far off base? If so, what's your take?
"It is dark. You are likely to be eaten by a grue." -- Zork
break in to Debian, it was notices within 24 hours. Break into Gentoo, noticed in 1 hour. Break in to Microsoft, not noticed for MONTHS.
To those who aren't intentionally trying to troll.. and computer journalists;
Yes, Linux servers can be compramised.
No, the sky is not falling.
No, it's not the end of Linux or open source.
Luck favors the prepared, darling.
Now consider what would happen if the Windows update service was compromized and hackers managed to get past Microsoft's tight security. These update servers could be used for WMD's (Windows Massive Disruptions)...
From excellent karma to terible karma with a single +5 funny post...
Leading to the hacking machine? Fixing the compromises on major linux servers is one thing, but why has nobody mentioned finding the perpetrators?
Anything in these logs on the source of the hacks? Probably another hacked machine, but perhaps it can be traced to a source.
Also, in any package that were compromised or attempted at, what is being inserted? Perhaps we can use it as a honeypot to catch a hacker?
Perhaps 2.4.23 should have a kernel allowance for a log that tells when somebody was trying to use the =2.4.22 exploit (or does it)?
no, its news because a very popular linux dist has been hacked which could effect a lot of people. that = news
damn microsoft bashing wannabee
I'd just set up a remote box specifically for logging and connect it to a cheapo dot-matrix line printer and have the logs printed to paper. Yeah, you might use a bunch of paper, but it also might come in real handy if you ever need to figure out what really happened to your box.
Someone you trust is one of us.
Does anyone have an old, cached copy of the DNS record for rsync.gentoo.org?
.
Diff it against what's out there now and we're only a quick trip to http://arin.net/whois from knowing who it was . .
-Peter
There is no sure thing in security but there is a simple step to make things a bit more reliable for logging.
If you really have a serious system where you want detailed logs you keep the logs for that system off that machine. Sure the machine that is logging could have been comprimized as well but that is twice as much work. Now you have to hack the machine but also the logger to erase the intrusion event.
In fact one of the things I've seen done is that events are logged on the machine and the logger. The idea was to provide not only redundant logging but also provide a front for hackers. A hacker would see the local logs and be too busy doctoring up those logs to check to see if there is an external logger.
In any event, the logging Gentoo did looks complete enough. They claim only 20 users did a sync against the server during the hour it was online and comprized.
PS, full props for the Lebowski quote!
Luck favors the prepared, darling.
Why not?
You take the keys of the developers [or even a cvs key] and then sign all the emerge files. There are only like 2000 new ones a day so at about 50ms a signature [for a really slow box] that's only 100 seconds of time [two minutes not much].
That way if the end user downloads compromised emerge files they could detect them.
Damn... I'm like a genius.
Someday, I'll have a real sig.
One of the servers that makes up the rsync.gentoo.org rotation was compromised. This box is not an official Gentoo infrastructure box and is instead donated by a sponsor. The box provides other services as well and the sponsor has requested that we not publicly identify the box at this time.
While it may run Gentoo, it is not stated as such, and could be very well be something else.
Get a free ipod.
To correct a few misconceptions in the previous comments.
It was not their server that was compromised, just a third party server in a round robin rotation. They don't own it, they don't maintain it - just someone else who donated server space.
The primary or master server is not accessible to users, it was not compromised, and so none of the original source files had a chance to be changed.
Only the 20 users that synchronized to this server even have a tiny chance of getting bad files. Having everyone sync now that this server is out of the rotation will immediately fix the problem.
Full disclosure 24 hours later. I give them a lot of credit for such a quick response and disclosure. This is very, very minor.
~J
Only 20 people sync'd with this server within that hour it was compromised...not a big deal, expecially when the compromise did not touch the portage tree and was mearly a rootkit install and some logs edited...not to mention it is a donated server used for other purposes, the attacker might not of even known it was used for Gentoo rsync...
But the server is down and will be scrubbed and re-sync'd, just to be safe
"Some things have to be believed to be seen." - Ralph Hodgson
facts are tricky like that:
"We don't know how many total servers the numbers were gathered from or what percentage of those servers is Linux vs. Windows, etc. It is safe to say that these results are true for the servers they monitor, but the percentages may not be true for all servers across the globe."
while there certainly exist a large number of linux machines that have been compromised, i can't imagine the number of infected linux machines is anywhere near that of the win32 systems infected by blaster/welchia/code red/nimda/sql slammer/klez/dumaru/sobig/etc. in the same time frame. i suppose the counting in this case depends quite a bit on the counter's definition of "compromised."
So I've been lurking around here long enough to spot certain trends. (Warning: generalizations ahead)
OSS advocates love to hate Windows
OSS advocates gloat when a new hole turns up in Windows
OSS advocates point to the number of worms, virus, etc in Windows and say, "Never us"
Then several OSS distros have a security breach in a short space of time.
OSS advocates respond with "Must be a conspiracy against us by some evil entity", "Hey, look how quick we caught it", "It would have been much worse with Windows".
Time to face facts gents. Windows is attacked FAR more than OSS. Why? Well, yes, it is full of holes. But downtown Philly is riddled with abandoned houses with no locks on the doors but they never get broken into. Why? No value in doing so. Not enough damage, headlines, misplaced glory, etc. But the main reason is that it is the dominant OS out there. I fear that we will see more and more attacks against OSS with it's growing popularity. If we all get our wish and 'nix takes over Windows dominant market position and is running on 90% of desktops, you will most likely find it a target for constant attacks like Windows has now.
We all know in order for 'nix to make it to the desktop, it has to become WAY more user friendly. Can't have Grandma trying to recompile the kernel now can we? User friendly unfortunately translates into users being able to do things that comprise security. Like opening attachments, downloading Trojans, etc. Then the great security built into the OS goes right out the window. no pun intended).
So before you all start crying about conspiracies, et al, just remember that we all may be victims of our own push to make the 'nix stuff more popular. By bragging about how secure it is, we just may be attracting the type of attack that is more sophisticated then the script kiddies attacking Windows. I imagine it's cool to brag to your friends that you broke into a Windows box. I imagine it's much cooler to brag that your rooted a Linux distro. Badge of honor and all that.
The opinions expressed here are not mine, but those of these dang voices in my head.
I though the Gentoo Zealot response would be:
:D
"Ah, but Gentoo's root exploit was compiled from source, so Gentoo got rooted 0.000000124% faster than Debian!"
Ah well, I like Gentoo myself. It is quite fun.
Fellowship 9/11
rooted 1% faster than a binary install!
With apologies to Torne, from whom I stole this quote.
I am one of many. My idea is not unique, nor do I expect my voice alone to sway you. I speak in a chorus of opinion.
IDS is placed on a system to follow an attack. Audit trails on sensitive machines reveal all commands executed, to the detail you desire.
Here is the point. Bruce Schneier says that the important part of security is not that you were compromised, but rather that you can react within a time frame to keep the damage to acceptable levels. If you can tolerate having your system compromised for weeks, don't invest in a lot of security. The short response time (2 hours at 11pmEST) here indicates that the Gentoo administrators care about responsiveness enough to check on it frequently.
When the CVS gateway to Bitkeeper on the Linux Kernel was compromised, the developers of Bitkeeper were able to show that they care enough about security that they invested in many checks and balances that caught the error immediately. Since then, Bitkeeper developers, interested in protecting their good reputation (which is VERY difficult to replace), are considering even more drastic measures.
As a bonus, some cracker spent a good few days or weeks writing this exploit. We get to keep it and deploy the solution with little hassle. And the compromised system, because good security practices are in place, was mitigated to minimize damage.
Read Schneier's book Secret and Lies to find out how security is really a process. Yes, I know it's a plug, but I just thought the book hit-home to the real point - "When, not if" you get compromised.
Several other posts here hint that the world will think less of Linux for this. False. True CIOs should see that Linux has the tools to completely identify and contain attacks. Every CIO knows attacks cannot be stopped, but rather they must be contained to acceptable levels.
So anyways, they did not count (most) worm incidents, as they would happen on non-server windows machines.
That does not mean that Linux boxen should not have better default security settings, of course.
Stephan
Let's face it, no OS is 100% secure. Operating Systems that are more secure than others still need to be on their toes. One security exploitation on a Linux box can still be as dangerous as a thousand (an underestimated ratio I'm sure) exploitations on a Windows box. However, I will take the body of security knowledge surrounding an OS to be as valuable as the initial security design principles in the OS in the first place; with that in mind, many Open Source OS's come out looking pretty good. I trust the Linux community to grind down and fix security problems and not sit around and emphasize the numerous security in a Microsoft product. If you're concerned, then help out developers by testing the software and reporting bugs. You could even code a few patches yourself, that being the whole point of community-based development.
Whether or not there is a deep and dark plot to root big Linux boxes is irrelevant. This is another opportunity to demonstrate the Open Source community's response to security issues to the rest of the computing community. If the heat is really on and this is not just another artifact of news gatekeepers getting over-zealous on a trend, then so be it. It is an opportunity to review and evolve Linux's security as well as the security processes that surround it.
One of the things I admire most about Linus Torvalds is his steadfast commitment to the quality of his product. It is a commitment that is focused on constant improvement, not PR damage control. I'm sure the real security guru's are sitting with a bit more comfort knowing their servers are running Linux.
Disclaimer: This post contains no constructive content whatsoever, swallow two tablespoons of salt and call me in the morning.
... did whoever did this steal any of our source code?
You'd think but www.openbsd.org doesn't run OpenBSD. Here's a link that explains why.
In fact, just last year ftp.openbsd.org did get compromised!
A radio maverick jumps to internet only. The Future of Rock n Roll
If you buy the idea that spammers are behind many of the recent worm/viruses, designed to turn machines into spam zombies, it's also probably reasonable to conclude that Windows isn't exactly a reliable platform to trojan; I'd bet a lot of trojans fail to infect properly simply due to Windows problems. And then there's the problem AV software, many of the machines being behind firewalls/NAT and being unreachable. And then there's people turning their PC's off when they don't use them.
OSS machines, however, are a much more reliable computing environment, meaning that any trojans are actually like to work, and work well. And I'd also wager that many OSS machines are used AS firewalls or bastion machines, and if compromised are easily accessable for spamming or use as stepping stones to other machines. And many of these machines are always on -- you don't have to worry about lack of reliability from disabled machines.
This makes more sense to me than any other conspiracy.
http://savannah.gnu.org/statement.html
On December 1st, 2003, we discovered that the "Savannah" system, which is maintained by the Free Software Foundation and provides CVS and development services to the GNU project and other Free Software projects, was compromised at circa November 2nd, 2003.
Since Gentoo doesn't have a "THE" source code repository, I'm afraid you've got some facts to get straight, Herr Coward.
The mirror had read-only rsync access to Gentoo's primary (US) mirror. Even if the tree were compromised, the changes could not propagate into the main tree. For that, one would require CVS access to the CVS repository, against which the primary rsync server is synchronized.
This was only posted as a matter of keeping our user community, and the OSS community as a whole informed.
Also, I believe the announcement gave mention of it, but the Portage tree on the primary mirror was re-created from the CVS repository immediately upon being notified that a mirror was compromised. Within 30 minutes, every Gentoo rsync mirror had a fresh copy of the tree automatically (as stated by Gentoo rsync mirror policy, mirrors are updated every 30 minutes in order to remain on the official rotation).
Sorry for the confusion, all, but there's really nothing to see here. But it was good clamouring practise for when/if a real Gentoo server is compromised. ;)
BD Phone Home!
Shameless plug. Like you weren't expecting it.
There are several methods with which you can gain access.
/etc/passwd by passing certain variables to a script run on a webserver, and from there you can attack user accounts. Also certain scripts have the error of allowing you to run system binaries, which again can be exploited. Always run your database daemon and your httpd in a chroot environment, with minimal access to system binaries, and always as a non privileged user.
1. Buffer overflows, or out of bounds issues, with services running on a server, eg ftpd, httpd, sendmail, bind (dns). This is where it is discovered to be possible to send malformed data to a service which the service is not expecting and wont deal with naturally. This sometimes results in the ability to send it some executable code which is read straight into memory and executed. Very easy to code around, very easy to detect, fairly easy to detect and very easy to exploit. This is the sort of attack that normally occurs against MS Windows et al, although sendmail, bind and various ftpds (wu-ftpd) have a reputation for being full of them.
2. Password sniffing. This is where someone sits between a user and their box and sniffs network traffic, etiher getting a password unencrypted (normal ftp login, pop3 etc etc) or a weak hashed. Fairly easy to do, and you have a login to the system when you do. Not normally seen these days as ssh is used, and you should always have a seperate restricted user login for other services which do not encrypt passwords (imap, pop3, ftp etc).
3. Issues with web scripts, that sometimes allow you to insert data into a database which the owner doesnt want you to do (or get a copy of his database) via SQL Injection attacks. Also it has been fairly common in the past to be able to get a copy of
The biggest problem these days is that a lot of services run as root, because they need to to bind to ports lower than 1024. This was done so it allows you to "trust" services on those ports as being proper ones, rather than ones run by a normal user. A way around this is to run all services as a standard user, on port ranges above 1024 and bound only to IP 127.0.0.1. This means that your services are no longer on the standard ports, but you can get around this by using ipfilter, pf or another port fordwarding tool to forward all traffic on external priviledged ports to the services on 127.0.0.1, allowing you to run services as non priviledged users while retaining compatability with the outside world.
It is VERY difficult to secure a server to near 100% levels, although you can get pretty close if you want to constantly be working at it. The goalposts change rapidly from day to day, and it can be hard to keep up. If you only run the services you really need, in chroot environments, and ensure that those services are well known services (apache for httpd, exim postfix or qmail for smtpd, pure-ftpd or pro-ftpd for ftpd, DJBDNS or bind 9 for dns) then you can be assured that there are trusted people looking at the source for exploits to fix as well as the untrusted people doing the same to exploit.
Good logging firewall rulesets, an IDS (intrusion detection system), and a remote logging facility are all plusses in the fight.
Cars are built out of steel, not glass. Glass is a very strong material. But hit it with a hammer and it shatters. Steel just gets dented.
Gentoo had "ductile" security. They were able to limit the damage because they had some kind of Tripwire/mtree-like program running on the inside. Given the speed of the response, my guess is that they had a response plan ready to go.
The lesson is that measures to limit the damage from a break are as vital as measures to prevent breaks in the first place. Fire prevention doesn't substitute for sprinkler systems, and intrusion prevention doesn't substitute for backups. You've got to have both.