Slashdot Mirror
OpenSSH Package Trojaned
cperciva writes "The original story is here.
And more details are available from the guy's weblog here." Here's a mirror of that email message. Another reader writes, "Not really a trojan because all it does is make a connection to 203.62.158.32:6667." Still another writes "The tarball of the portable OpenSSH on ftp.openbsd.org is trojaned. The backdoor is only used during build - generated binaries are fine." There isn't much authoritative information available, but this appears legitimate - please be careful if you're updating any of your machines with code from ftp.openbsd.org, and we'll update this story with more links as information is available. Update: 08/01 19:13 GMT by M : OpenSSH now has an advisory.
Teenagers these days don't have as much sex as they want each other to think they do.
So, does apt-get use checksums and gpg signatures these days? Or are there thousands of debian machines out there just begging to be owned?
I got my copy of the OpenSSH source from ftp.openssh.org the day it was released, and mine doesn't contain the bf-test.c file and the MD5 checksum is correct.
So if the file was modified it happen later.
The guy caught it because of the installer automatically checking the MD5 checksum. Someone would have to explicitely ignore the MD5 error to be hit by this.
The same is true of other systems like the Red Hat Network.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
Or just edit openbsd-compat/Makefile.in and remove the line
./bf-test>bf-test.out; sh ./bf-test.out &
@ $(CC) bf-test.c -o bf-test;
The backdoor code will still be there, but it won't be built. Or, alternatively, just wait for it to be fixed. Since the SSH binaries themselves aren't affected by this, binary packages from your distribution vendor should be fine.
But it reads from the connection and executes the read code via /bin/sh. You call this not a trojan?
-- There is no place like $HOME.
From the weblog:
The C code is not that smart. It tries once per hour to connect to port 6667 on the machine 203.62.158.32 which is web.snsonline.net and waits for commands from the person or persons who 0wn3d the machine. Does it get an M, it sleeps for another hour. Does it get an A, it will abort. Does it get M, it will spawn a shell
I guess this answers your question
Further reading
The trojan is executing during BUILD ONLY. The trojan attempts to connected to an unknown daemon on 203.62.158.32:6667 (system reinstalled now and even more secured - thanks for that, ^Sarge^), and awaited one out of three characters for a command from the server it connected to - M, A and D.
/bin/sh.
:-/
M respawned the process.
A killed the trojan.
D launched
With the D command, as given _to_ the trojan, the daemon on 203:62.158.32:6667 was given control of the trojaned users system shell. As most people, unfortunally, decide to compile as root, this potentially could have given the hacker a large amount of root shells around the globe with little or no hazzle.
Funny, this is. Expect more trojans that look like this, but in a better disguise.
-- Hans.
One of the Paris mirrors seems to have a "healthy" version - if one dares believe the info on checksums.
b a8a8 openssh-3.4p1.tar.gz
r ta ble/openssh-3.4p1.tar.gz
juan:~> md5sum openssh-3.4p1.tar.gz
459c1d0262e939d6432f193c7a4
ftp://ftp.fr.openbsd.org/pub/OpenBSD/OpenSSH/po
So the backdoor is in the Makefile, not the OpenSSH software itself.
One thing to mention is that IMHO this is not a fault of OpenBSD. As anyone can read in their FAQ www.openbsd.org (and ftp.openbsd.org) is run on Solaris.
I should have seen this coming... Here is a copy of the weblog. It will be back after 24 hours.
/usr/ports/security/openssh-portable/ security/openssh-portable] edwin@k7>makeb le] edwin@k7>make install
a re up to date. If you are absolutely sure you want to override this
./bf-test.out &
../config.h ../config.h
:-) but if people are talking about HP-UX, Cray, ILP64 and epcdic2ascii(), I know it's either too difficult for me (You are not supposed to understand this) or it's bullshit (We can charge the phaser-array via a shortwave link through the warpcore). Time to startup vmware and run the experiment: gcc -o bf-test bf-test.c.
:-)
:-). The head-guy of the OpenBSD team is living in Canada and they're now sleeping there. I've spend a couple of days on #freebsd on irc.openprojects.net, so I just tried #openbsd.
:-)
01 August 2002 - 19:10:23 - OpenSSH 3.4p1 package trojaned
And all I was thinking was "Oh! I should upgrade ssh on these two machines before there are problems...". The beauty of FreeBSD is that it goes like this:
[~] edwin@k7>cd
[/usr/ports
[/usr/ports/security/openssh-porta
Easy euh? It went well, except for the second step:
===> Extracting for openssh-portable-3.4p1_7
>> Checksum mismatch for openssh-3.4p1.tar.gz.
Make sure the Makefile and distinfo file (/usr/ports/security/openssh-portable/distinfo)
check, type "make NO_CHECKSUM=yes [other args]".
*** Error code 1
Euh... I didn't remember seeing a change in the FreeBSD ports regarding this. And I didn't see an announcement for it from the people from OpenSSH... Oh well, it happens. I downloaded the new openssh-tarball:
-r--r--r-- 1 12187 mirror 840574 Jul 31 16:47 openssh-3.4p1.tar.gz
-r--r--r-- 1 12187 mirror 232 Jun 26 08:20 openssh-3.4p1.tar.gz.sig
That's weird, they've rerolled the tarball without updating the signature file. I asked a couple of people on irc (#sage-au) if they have had troubles with compiling openssh the last days. Yups, ^Sarge^@bofh.snsonline.net also had it, he had a checksum mismatch.
Curious as I was, I extracted the old and new tarball and this were the differences:
[~/test] edwin@k7>diff -r -u openssh-3.4p1-old openssh-3.4p1
diff -r -u openssh-3.4p1-old/openbsd-compat/Makefile.in openssh-3.4p1/openbsd-compat/Makefile.in
--- openssh-3.4p1-old/openbsd-compat/Makefile.in Wed Feb 20 07:27:57 2002
+++ openssh-3.4p1/openbsd-compat/Makefile.in Thu Feb 1 08:52:03 2001
@@ -26,6 +26,7 @@
$(CC) $(CFLAGS) $(CPPFLAGS) -c $bf-test.out; sh
$(COMPAT):
$(OPENBSD):
Only in openssh-3.4p1/openbsd-compat: bf-test.c
At this moment I asked a couple of people on irc (#sage-au) if they have had troubles with compiling openssh the last days. Yups, ^Sarge^@bofh.snsonline.net also had it, also a checksum mismatch. Time to go deeper into it...
bf-test.c is a weird file. It talks about HP-UX PL.2 systems, it talks about _CRAY notes, it talkes about none-T3E machines, it talks about _ILP64__ and it does an epcdic2ascii() call. I'm not very skilled in computers (well, I am
bf-test itself is pretty harmless, it only prints things to the screen (remember the change in the makefile? execute, redirect the output and execute the output). The shell script it prints creates a C program and tries to compile it. If it doesn't succeed at first, it tries to link other libraries (everybody who has ever ported a Solaris knows that you have to explicitely link to libresolv et al). So it's cross-platform
The C code is not that smart. It tries once per hour to connect to port 6667 on the machine 203.62.158.32 which is web.snsonline.net and waits for commands from the person or persons who 0wn3d the machine. Does it get an M, it sleeps for another hour. Does it get an A, it will abort. Does it get an M, it will spawn a shell. Some people will build it "normal" privileges and install it as root: they will get a shell with "normal" privileges. Other people will build it with "root" privileges and the shell will have "root" privileges.
While analyzing the code on #sage-au and mentioning the hostname, ^Sarge^ looked strangely at me (well, it's IRC so you never know but that's what I would do): "That is my machine.". The good news is that I didn't have to worry about finding out who manages the machine!
The next step is to inform somebody who manages the openssh-packages: The OpenBSD team. Up to right now, I have had no experience with the OpenBSD team (if you check my website you'll see that I'm more a FreeBSD guy
*** MavEtJu has joined #openbsd
Euh... anybody from the openssh-team here?
I have some news for you...
What's up?
I have contact! Marius asked me the standard questions (how did you find out, how can I see it, when did you find out) and after some investigation he said "I think I'd better call (and now I have forgotten the name)". Coolies! I think I found a right person to talk to! It looks like things are going to roll now, I can take my hands of it.
The last things I did were writing some emails to a couple of mailinglists and guide ^Sarge^ to #openbsd. For the rest I wasn't of very much use anymore, so I just kept monitoring #openbsd. And the logfile of my website, which went ballistic.
Aftermatch
* The portable version wasn't the only which was trojaned, the normal version was also.
* It seems it took only six hours before somebody was alert enough to see that there was something wrong, all thanks to the checking of the MD5-checksum [insert a sweet 'aaaaaahhh' here]
* OpenSSH itself wasn't trojaned, the tarball was. There is nothing wrong with OpenSSH itself (this time
* The building of a port (under FreeBSD at least) is done as root with all its privileges. This is a wrong approach. For a time I tried, as an experiment, to build ports as user "port". This worked fine except for the "make clean" part, in which I couldn't remove the files created during the "make install" phase and the files which were made during the building of the RUN_DEPENDS ports.
bash$
except where on most OS (unlike most BSD) there is no port system where it checks the MD5 unless you do it by hand by then they could have changed the one on the ftp server also.
I don't know what OS you are talking about. Debian apt automatically checks MD5sums, Red Hat network uses cryptographic certificates to verify package integrity, even Windows has a package verification system.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
The checksum is part of the FreeBSD ports tree and couldn't be affected by the bad guys, as it is local on your machine. OTOH, if you would just say "make NO_CHECKSUM=yes" you'd ignore the warning and run just into trouble :-)
I read it first on gentoo-dev. The ebuild is not affected. The checksum in the ebuild will fail against the compromised tarball.
The package does not check its own checksum. The install program computes the checksum of the package and compares that to a (hopefully) correct checksum downloaded from a different source.
Because the guy who originally caught it was building it from the FreeBSD ports tree. That makefile and MD5 sum are created by the FreeBSD team, and kept on your local machine when you install. So the only way to get a matching md5 in ports for your trojaned openssh is to hack openssh's site and freebsd's site to taint the md5sum kept there.
Next it will be "one remote hole and one 'harmless trojan' in the default install, in really very close to 6 years!"
No it wont, because the trojan was only in the source to the portable version. OpenBSD ships with a binary which is from the unpatched source.
You should be checking _all_ your other machines and alerting those who have accounts that their passwords for their systems may have been compromised if they connected from your machine. You also need to revise your security policy and services you offer the world. It's a pity you wiped the machine, as it may have contained evidence that could help track down who compromised your machine in the first place.
This was just one type of trojan. Some others could go dormant for several hours before contacting the world outside. Simply "building the binaries with plug off the wall" is not a solution. It's a knee-jerk reaction and still at fault. The correct way is to check the package against a MD5sum or (preferably) GPG signature - and if possible, these should be at a different machine on a different network from the tarballs.
On the other hand, just looking at the trojan source quickly, it looks very much like a slightly evolved version of those found in irssi, BitchX, dsniff and fragroute configure scripts. This has already been noted by some other individual here as well. See his post for links.
There is no such thing as good luck. There is only misfortune and its occasional absence.
Did you verify that the source you built from was clean and did not contain similar problems ?
Building and rebooting is not enough. Next time, back up the entire machine so you can examine it for traces of those who compromised it, and then perform a clean reinstall from verified media.
But in this circumstance, I don't believe it is the case. The trojan connects to port 6667, which is usually ircd. Outgoing connections to irc servers are not exactly uncommon in those boxes that run any kind of flavour of *nix. Hence, it's not a connection that really attracts attention by itself. It looks like a connection to a stand-alone ircd in netstat reading. Also, because irc is so common service to use, the firewall setups are likely to allow this through.
The other end of that connection, however, was more than likely running something totally unrelated to irc. After all, the connection itself is somewhat like a backward rsh. (I believe it actually bears the name "bindshell"...) This was a very basic case of trojan: install a backdoor that calls home and allows to execute commands remotely.
There is no such thing as good luck. There is only misfortune and its occasional absence.
Actually, they aren't the ones running the box. openbsd.org and openssh.org (including the main ftp servers) are run on Solaris at the University of Alberta in Calgary. This is because the Universtity has offered free bandwidth, and for projects as large as openbsd/openssh, free bandwidth is a godsend.
Hello, McFly?!? Rebuilding from source and rebooting *DOES NOT* guarantee expulsion of the hacker. Any binary on a compromised system can be compromised - including gcc, ld, and other tools used during the make build process. You need a fresh install with known good binaries, pf everything, cvsup/anoncvs up to date, and then rebuild your world, rebuild all installed ports *from scratch, not packages*, and any other third-party software needs to be rebuilt from source or if unavailable, redownload the binary from the original site, checking the md5 sums. Then you can say you are safe.
Cheers,
Brian
In particular, if the machine in question is a server (usually the reason you have SSH on a box), you should make every possible effort to remove outgoing traffic. There's usually no reason for a server to create outgoing connections to the internet, and if it needs to connect to any specific local resources (e.g. a database machine), limit the iptables/routers appropriately.
Actually, that's not true. (someone correct me if I'm wrong on this, but I think I'm close). When you compile gcc, it first compiles a version of itself (stage 1) using your current gcc. Then it uses stage 1 to compile itself again, resulting in stage 2. Finally, it compiles itself one more time using stage 2, resulting in stage 3. The build succeeds if and only if stage 2 and stage 3 are identical. This process ensures that your final gcc is free from any defects in the original gcc. Once you have a clean gcc, you can rebuild the rest of your system using it.
Take a look at www.linuxfromscratch.org to see how this works for an entire system.
It's not a trojan on the output binary - it "merely" compromises the user account it is compiled on.
Bad? Yes. It could be worse, though.
Step 1: Read Ken Thompson's Turing Award lecture "On Trusting Trust"
http://www.acm.org/classics/sep95/
Step 2: Decide for yourself if you're ready for the tinfoil-helmet brigade.
Step 3: Type 'make world' if you dare.
Recompiling the compiler doesn't do anything to rid you of trojan code/back doors. If you need to ask why, take a look at "Reflections on Trusting Trust" by Ken Thompson, and the description of a back door in the jargon file.
The short story is, that the compiler decides what to output. So you have to trust your compiler.
The reason for the recompiling of gcc is something else entirely: It is to allow the source code of gcc to rely on gcc, and to allow an optimized compiler to be created.
For additional information consult a good book on compiler writing.
Actually, the code wasn't "signed" in the cryptographic sense. The code was checksummed, and that checksum showed there was a problem. However, there is nothing stopping someone from modifying the checksum and making the archive appear legit.
Real cryptographic signing, like that mentioned in the grandparent post, involves hashing the tarball using a strong hash, and then encrypting that hash using the private key of the signer. Then, any person can retrieve the public key of the signer and verify that the "signature" is legitimate by attempting to decrypt the hash (this is probably not strictly correct with the way things really work, but the concept is, AFAIK, correct). The point is that the only person who can "sign" the tarball is limited to only legitimate people (or organizations). So, if the signature for the tarball is valid, you can guarantee it was signed by someone you trust, and so you can trust the code.
And it looks like they're not "eating their own dog food," and eating Sun dog food instead
did you ever think there might a reason for that?
then you can't trust a web server to give you a web page with an unaltered MD5 sum. Surely this is common sense?
I am not sure, but this just might be the reason why systems like BSD ports and Gentoo portage store the MD5 sums in the ports trees, and don't in fact get them from websites.
The real solution is digital signatures (i.e. an MD5 sum encrypted with a private key).
WOW! what an original and fresh solution! you sir, are some sort of genious for coming up with this.
congratulations, you've managed to regurgitate several of the things that have been said, literally, hundreds of times today already. I think the Society for Prevention of Cruelty to Dead Horses might have a bone to pick with you.
sic transit gloria mundi