Slashdot Mirror
Gentoo rsync Server Compromised [updated]
costela writes "LWN points out that the Gentoo project
fired out an alert about one compromised rsync server." From the message itself: "However, the compromised system had both an IDS and a file integrity checker installed and we have a very detailed forensic trail of what happened once the box was breached, so we are reasonably confident that the portage tree stored on that box was unaffected." Update: 12/03 22:54 GMT by T : One more damage report: gibson writes "The Free Software Foundation recently discovered that its software host site was compromised a month ago. The compromise appears to be the same as the recent attacks on the Debian servers. The site is shut down until Friday while they install replacement hardware and verify the authenticity of the hosted source code."
This is the sort of site that gets /.'d so here's the full text.
Just to summarize - they don't know how it was done but they're pretty certain no damage was done.
Text
On December 2nd at approximately 03:45 UTC, one of the servers that makes up the rsync.gentoo.org rotation was compromised via a remote exploit. At this point, we are still performing forensic analysis. However, the compromised system had both an IDS and a file integrity checker installed and we have a very detailed forensic trail of what happened once the box was breached, so we are reasonably confident that the portage tree stored on that box was unaffected. The attacker appears to have installed a rootkit and modified/deleted some files to cover their tracks, but left the server otherwise untouched.
The box was in a compromised state for approximately one hour before it was discovered and shut down. During this time, approximately 20 users synchronized against the portage mirror stored on this box. The method used to gain access to the box remotely is still under investigation. We will release more details once we have ascertained the cause of the remote explo it.
This box is not an official Gentoo infrastructure box and is instead donated by a sponsor. The box provides other services not related to Gentoo Linux as well and the sponsor has requested that we not publicly identify the box at this time. Because the Gentoo part of this box appears to be unaffected by this exploit, we are currently honoring the sponsor's request. That said, = if at any point, we determine that any file in the portage tree was inappropriately modified, we will release full details about the compromised server.
Again, based on the forensic analysis done so far, we are reasonably confid= ent that no files within the Portage tree on the box were affected. However, t= he server has been removed from all rsync.*.gentoo.org rotations and will rema= in so until the forensic analysis has been completed and the box has been wiped and rebuilt. Thus, users preferring an extra level of security may ensure that they have a correct and accurate portage tree by running: emerge sync Which will perform a sync against another server, thus ensuring that all fil les are up to date.
Aide-toi, le Ciel t'aidera - Jeanne D'Arc.
remote logging
The kernel exploit was a local one. Evidence has shown that the exploit used against the rsync server was remotely exploited. Good thing that if anything had happened (which nothing has shown up yet, according to the #gentoo-dev channel on FreeNode), it would have only affected about 20 users. Still sucks that there's an exploit at all though.
"You tried your best and failed miserably. The lesson is...never try. Heh!" -Homer
By "file integrity checker" I presume they mean something like AIDE.
One makes hashes of each file and stores them on a non-networked system and/or read-only media. Then periodically runs a check (hopefully from a statically linked binary that is also on RO media) on the files and compares the hashes.
If they match (and any number of other conditions are met, like the machine and the media the hashes were stored on are physically secure, etc.) you can say with reasonable certainty that the files are unmolested.
-Peter
OK, I RTFA and it doesn't look to me like they were going after Gentoo specifically. The way I read it, this was just a box somewhere that a sponsor had set up to house a Gentoo rsync node, and had a bunch of other stuff on it as well. The box got rooted and the cracker didn't touch any of the other stuff on the box -- just what he needed to obfuscate his entry and do all the usual rootkit stuff.
Why do they do this? Because they can. Personally, I blame that darn rap music.
Someone you trust is one of us.
Uhh...it's a round robin. An rsync rotation server was compromised remotely, and one suid file was found. This was not a DNS hijacking or anything like that.
"You tried your best and failed miserably. The lesson is...never try. Heh!" -Homer
There is no sure thing in security but there is a simple step to make things a bit more reliable for logging.
If you really have a serious system where you want detailed logs you keep the logs for that system off that machine. Sure the machine that is logging could have been comprimized as well but that is twice as much work. Now you have to hack the machine but also the logger to erase the intrusion event.
In fact one of the things I've seen done is that events are logged on the machine and the logger. The idea was to provide not only redundant logging but also provide a front for hackers. A hacker would see the local logs and be too busy doctoring up those logs to check to see if there is an external logger.
In any event, the logging Gentoo did looks complete enough. They claim only 20 users did a sync against the server during the hour it was online and comprized.
The rsync servers sync with...rsync!
So, yes, changes in the source rsync tree would trickle down to all the mirrors. It wouldn't actually compromise those servers, in the root-on-the-box sense, but they would be serving compromised data.
Anything is possible given time and money.
To correct a few misconceptions in the previous comments.
It was not their server that was compromised, just a third party server in a round robin rotation. They don't own it, they don't maintain it - just someone else who donated server space.
The primary or master server is not accessible to users, it was not compromised, and so none of the original source files had a chance to be changed.
Only the 20 users that synchronized to this server even have a tiny chance of getting bad files. Having everyone sync now that this server is out of the rotation will immediately fix the problem.
Full disclosure 24 hours later. I give them a lot of credit for such a quick response and disclosure. This is very, very minor.
~J
NetCraft reports Linux and Apache (Red hat version). http://uptime.netcraft.com/up/graph?site=rsync.gen too.org
Fortress of Insanity
Only 20 people sync'd with this server within that hour it was compromised...not a big deal, expecially when the compromise did not touch the portage tree and was mearly a rootkit install and some logs edited...not to mention it is a donated server used for other purposes, the attacker might not of even known it was used for Gentoo rsync...
But the server is down and will be scrubbed and re-sync'd, just to be safe
"Some things have to be believed to be seen." - Ralph Hodgson
He's trying to figure out what unnamed company provided the hacked box.
Common sense is not so common.
Simply put, Gentoo didn't own the machine, there are lots of "rsync" servers that people use (like web sites, but used for downloading files only). Because the people who run gentoo don't own the server, they aren't responsible for it. All this leaves me with a very uncomfortable feeling. I have some websites running on linux servers (not mine) from rackspace providers. Should I be worried ? No! Gentoo is really a desktop OS. Your websites will not be affected. In fact from the sounds of things no one will of been affected, just the 1 server.
This one's been in development for a while, and will be going live soon probably. Read GLEP 14,
remote logging
Do you mean that people don't use line printers any more???!!? Back in the good old days, (not really) we'd have the computer print the diffs of any files that ever changed on the system in real time!
So anyways, they did not count (most) worm incidents, as they would happen on non-server windows machines.
That does not mean that Linux boxen should not have better default security settings, of course.
Stephan
Is it not possible at all to secure a server ?
The old adage goes something like: the only safe computer is unplugged, encased in concrete, and buried at a radioactive waste site.
It sounds like the admins at this place were doing a good job, hence catching the break-in in 1 hour and having a log trail of what happened. The interesting thing will be when they find out the exploit used to get in. The Debian rooting caused a new kernel version, because the flaw was found to be in the Linux kernel. Hopefully we'll soon know what weakness the attacker used in this case, and another hole will be filled.
Security is never perfect. You worry based on your risk level. The defence department is not allowed to put any classified information on a public network for that very reason. This is why Slashdot panics over Internet-Ready weapon systems, and electronic voting: they put something significant or dangerous at risk. On the other hand, if it's just your blog and photo album, then are you worried about a compromise?
There's also exposure level. There are hundreds of people trying to break into Microsoft, or the DoD. The very best people will put their best effort into it. Smaller sites mostly get script kiddies. The tools they use are based on known exploits, which have probably already been patched. So unless you really offend someone, patching will keep most of us out of trouble.
This break-in will cause one more security flaw to be fixed, and the world will be a little bit safer, for now.
The server wasn't actually running Gentoo Linux from what I have read.
FuckTheFuckingFuckers.com - Post your th
Because, save for an attacker compromising all Gentoo workstations and altering the Portage application itself, this is not plausible. `emerge sync` updates only the tree of ebuilds - text file application install scripts, analagous to Makefiles. The process is quite similar to BSD's `cvsup` process. The only files modified in this process are contained in /usr/portage/ (or another location optionally configured by the user). The `emerge` program itself is contained in /usr/bin, and is not touched by the rsync process.
Sorry to tear that nasty gash in your tin-foil hat, though.
BD Phone Home!
Shameless plug. Like you weren't expecting it.
http://www.winnetmag.com/windowspaulthurrott/Artic le/ArticleID/41035/windowspaulthurrott_41035.html
During an oddly-underpublicized security Webcast Monday, Microsoft revealed that hackers subject the company to 2500 to 3000 electronic attacks every day, or over 100,000 a month. Yet despite this massive number of attacks, the last successful intrusion occurred over three years ago, during the infamous October 2000 security breach. But the software giant says the biggest security risk to the company isn't external electronic attack of its Web properties, but rather its huge fleet of mobile workers and partners--some 60,000 strong--that access the company's 175 remote access points on a regular basis.
We've taken a deep look inside Microsoft to see how we can improve security at every level," sad Mike Nash, the vice president of the Security Business Unit at Microsoft, during the Webcast. "A lot of the technology we use Microsoft applies directly to [customers'] work."
Microsoft revealed some other interesting statistics during the Webcast. The company uses Computer Associates' eTrust security management suite to secure its networks. It uses two-factor authentication (user name/password and smart card) to better secure its intellectual property.
"Sufferin' succotash."
It doesn't have, but would be trivial to implement. Here is my suggestion how a patch for that should look (untested):
Do you care about the security of your wireless mouse?
Not to criticize (and I am a Deb fanatic, so excuse if i get a little zealous), but the box wasn't gentoos'
:-D)
Don't get me wrong, they did a _FANTASTIC_ job catching the break-in, but at the end of the day, there's a good chance the IDS was installed by the people who OWNED the box, and not Gentoo proper.
While I'd hope and expect the box was running gentoo, there's no evidence of this, nor of it having been Gentoo's work that caught on.
I fully believe in full disclosure of break-ins (though a delay on revealing vuln's is fair in some MAJOR cases, it should still be done), and they did everything they should have.
Debian caught it within 24 hours, GNU within a month, MS's last breakin that i can recall, took months.
It's not about what OS, but the administrators.
If your admin doesn't take proper precautions for a highly visible boxen, then these things will happen.
Sure the breakin likely could've been prevented had they been uuber up-to-date, but really, the problem is inherent with OSS
Again, I disclaim. OSS is a Very Good Thing, BUT, because of this, vuln's are posted far more often than with CSS (no, not style sheets).
Because of that, the patch-turnaround time is much smaller with OSS, and we as admin's who love our linux/bsd/OSX boxen MUST stay on top.
I dont have a regimented update schedule for my personal box, even though I should, but because vulns' are usually found and widely publicized very quickly in this world, the users do have to take more precautions.
If nothing else, this indicates a need for a clearing house (other than ones that tend to be used exclusively by those that seek to find vulns), with links to patches, packages, etc for various distros could be a very good thing.
Security.debian.org is a good example, but as recent times have shown, even debian isn't perfect (as much as I tell others it is
Love your distro, tend to your boxen, but admin like you could be hacked tomorrow. As gentoo/gnu/debian/MS/*BSD have demonstrated in the past, it can and WILL happen to your box.
I still see regular hist on apache for old IIS vulns. People wouldnt try it if there werent still vulnerable people, and the worms couldnt try it if they'd ever been cleaned properly.
It's not about OS.
It's not about vulns.
It's not even about the hackers.
It's the admin's who defend these boxes.
So to the recent distros, Deb & Gentoo alike, I salute the admins who have done a great job keeping the public informed to potential problems. I only hope that when I finally get out of this bloody school, I'll be half the admin that these guys have working for them.
-- (appended to the end of comments you post, 120 chars)
There are several methods with which you can gain access.
/etc/passwd by passing certain variables to a script run on a webserver, and from there you can attack user accounts. Also certain scripts have the error of allowing you to run system binaries, which again can be exploited. Always run your database daemon and your httpd in a chroot environment, with minimal access to system binaries, and always as a non privileged user.
1. Buffer overflows, or out of bounds issues, with services running on a server, eg ftpd, httpd, sendmail, bind (dns). This is where it is discovered to be possible to send malformed data to a service which the service is not expecting and wont deal with naturally. This sometimes results in the ability to send it some executable code which is read straight into memory and executed. Very easy to code around, very easy to detect, fairly easy to detect and very easy to exploit. This is the sort of attack that normally occurs against MS Windows et al, although sendmail, bind and various ftpds (wu-ftpd) have a reputation for being full of them.
2. Password sniffing. This is where someone sits between a user and their box and sniffs network traffic, etiher getting a password unencrypted (normal ftp login, pop3 etc etc) or a weak hashed. Fairly easy to do, and you have a login to the system when you do. Not normally seen these days as ssh is used, and you should always have a seperate restricted user login for other services which do not encrypt passwords (imap, pop3, ftp etc).
3. Issues with web scripts, that sometimes allow you to insert data into a database which the owner doesnt want you to do (or get a copy of his database) via SQL Injection attacks. Also it has been fairly common in the past to be able to get a copy of
The biggest problem these days is that a lot of services run as root, because they need to to bind to ports lower than 1024. This was done so it allows you to "trust" services on those ports as being proper ones, rather than ones run by a normal user. A way around this is to run all services as a standard user, on port ranges above 1024 and bound only to IP 127.0.0.1. This means that your services are no longer on the standard ports, but you can get around this by using ipfilter, pf or another port fordwarding tool to forward all traffic on external priviledged ports to the services on 127.0.0.1, allowing you to run services as non priviledged users while retaining compatability with the outside world.
It is VERY difficult to secure a server to near 100% levels, although you can get pretty close if you want to constantly be working at it. The goalposts change rapidly from day to day, and it can be hard to keep up. If you only run the services you really need, in chroot environments, and ensure that those services are well known services (apache for httpd, exim postfix or qmail for smtpd, pure-ftpd or pro-ftpd for ftpd, DJBDNS or bind 9 for dns) then you can be assured that there are trusted people looking at the source for exploits to fix as well as the untrusted people doing the same to exploit.
Good logging firewall rulesets, an IDS (intrusion detection system), and a remote logging facility are all plusses in the fight.
I have received >2000 sobigs from >1000 IP addresses.
There's your 1000 rooted boxes, and I didn't even need to do it myself.
YAW.
Your head of state is a corrupt weasel, I hope you're happy.
There's more than one person behind this.
Nearly the same time that Debian's boxes were rooted, a
"friend of Debian" had his system rooted too, and the
exploits and rootkits used were very similar, as the
sysadmins compared notes. However, they were subtly
different and the most likely explanation is that the
two hackers knew each other, and exchanged some
information, but weren't the same person.
YAW.
Your head of state is a corrupt weasel, I hope you're happy.
... No modding up necessary.
emerge sync doesn't touch emerge. Basically, all emerge sync does is get a listing of the Portage tree and fetch the latest ebuilds, and delete whatever is old. The only thing emerge sync does in relation to emerge itself is tell you that a new version is available if there is one.
Why doesn't anybody talk about FreeBSD security lockdown levels? My friend was telling me something about this. Supposedly it's possible to prevent some files from being written to, even by root. And you cannot go to a lesser security level, you can only go higher (until a reboot).
There is supposedly some stuff out there that allows finer grained permissions, e.g. running a service as a normal user, but saying that it can create an open port of this number once and only once (that would normally require root privaledges). Sort of like setuid but for specific tasks.
Buffer overflows and holes will continue to happen. But we need to get to a stage where even if an attacker gets in, they cannot do anything. What is happening with SELinux?
I was going to post it here, but the moronic lameness filter won't let me. So you'll need to look at rsync.samba.org.