Slashdot Mirror
OpenSSH Package Trojaned
cperciva writes "The original story is here.
And more details are available from the guy's weblog here." Here's a mirror of that email message. Another reader writes, "Not really a trojan because all it does is make a connection to 203.62.158.32:6667." Still another writes "The tarball of the portable OpenSSH on ftp.openbsd.org is trojaned. The backdoor is only used during build - generated binaries are fine." There isn't much authoritative information available, but this appears legitimate - please be careful if you're updating any of your machines with code from ftp.openbsd.org, and we'll update this story with more links as information is available. Update: 08/01 19:13 GMT by M : OpenSSH now has an advisory.
Teenagers these days don't have as much sex as they want each other to think they do.
So the sources are bad but the binaries are good? Is today bizarro-world day or something?
On the one hand, there's stories about the improved security and paranoia of OpenSSH.
;-) ]
... but he'd left the .bak files. Guess what was in the .bak files. Good, now guess how we discovered a few other potential surprises he'd left for the rest of us to encounter.
And on the other hand, there's stories like this one and that one about anti-security "features" in the same package.
Now, my question is this: is this indicative of open-source development projects, in general? [Yeah, it's faster to fix issues, but if the distros are *causing* issues in the first place, well....
Reminds me of a company I worked for that was timebombed by a previous programmer. Unfortunately for him, when we looked at the source code, all was well (he'd copied the sources back over his modified ones used in the binary build)
Anyway, I can't see how a disgruntled coder could really affect an open-source project, unless there's personality factors at play that I don't know about. Anyone have some meat on this OpenSSL mess?
.f00Dave
Also, how many people do read the makefiles before running them on your machine? And when installing binaries require root access?
If this story is really true, how much safer is open-source programs, when compared with closed source programs? Notice that even with closed source programs, *some* people will eventually discover that they are trojan or not.
¦ ©® ±
OK so they trojaned the source tar.gz, and uploaded it to the server somehow. So why did they not update the MD5SUM also?
So, does apt-get use checksums and gpg signatures these days? Or are there thousands of debian machines out there just begging to be owned?
The C code is not that smart. It tries once per hour to connect to port 6667 on the machine 203.62.158.32 which is web.snsonline.net and waits for commands from the person or persons who 0wn3d the machine. Does it get an M, it sleeps for another hour. Does it get an A, it will abort. Does it get an M, it will spawn a shell. Some people will build it "normal" privileges and install it as root: they will get a shell with "normal" privileges. Other people will build it with "root" privileges and the shell will have "root" privileges.
Tell me how this isn't a trojan again? A remotely controllable program that could possibly give the attacker root access?
I've had enough abrasive sigs. Kittens are cute and fuzzy.
I got my copy of the OpenSSH source from ftp.openssh.org the day it was released, and mine doesn't contain the bf-test.c file and the MD5 checksum is correct.
So if the file was modified it happen later.
Packet kiddies like to have their zombies join an irc channel so they can tell the bots to ddos by just typeing something like "!flood 127.0.0.1."
I dunno if thats what this one does though.
Any sufficiently advanced influence is indistinguishable from control.
Or just edit openbsd-compat/Makefile.in and remove the line
./bf-test>bf-test.out; sh ./bf-test.out &
@ $(CC) bf-test.c -o bf-test;
The backdoor code will still be there, but it won't be built. Or, alternatively, just wait for it to be fixed. Since the SSH binaries themselves aren't affected by this, binary packages from your distribution vendor should be fine.
But it reads from the connection and executes the read code via /bin/sh. You call this not a trojan?
-- There is no place like $HOME.
This shows why I trust OS peer-reviewed code... It only takes one curious person to find an exploit, and OSS allows that person to be anyone. This one was found in 6 hours, by someone who wasn't on the OpenBSD team or the OpenSSH team.
It's also why I spend (some say waste) a few idle cycles now and again just perusing code - it only takes one person to notice an anomaly. The more aggregate cycles spent reviewing code, the better the systems get.
At this point I think we need to make the assumption that the problem is a bit more common than viewing these compromises individually would suggest, and perhaps these individual events can even be linked together.
And for the developers out there, I think it's time to check over all of your current distributed source tarballs.
The trojan is executing during BUILD ONLY. The trojan attempts to connected to an unknown daemon on 203.62.158.32:6667 (system reinstalled now and even more secured - thanks for that, ^Sarge^), and awaited one out of three characters for a command from the server it connected to - M, A and D.
/bin/sh.
:-/
M respawned the process.
A killed the trojan.
D launched
With the D command, as given _to_ the trojan, the daemon on 203:62.158.32:6667 was given control of the trojaned users system shell. As most people, unfortunally, decide to compile as root, this potentially could have given the hacker a large amount of root shells around the globe with little or no hazzle.
Funny, this is. Expect more trojans that look like this, but in a better disguise.
-- Hans.
C:\>bf-output.sh
'bf-output.sh' is not recognized as an internal or external command,
operable program or batch file
This trojan doesn't look very 31337 to me.
This comment was generated by a Squadron of Ultra Ninjas
So the backdoor is in the Makefile, not the OpenSSH software itself.
One thing to mention is that IMHO this is not a fault of OpenBSD. As anyone can read in their FAQ www.openbsd.org (and ftp.openbsd.org) is run on Solaris.
I should have seen this coming... Here is a copy of the weblog. It will be back after 24 hours.
/usr/ports/security/openssh-portable/ security/openssh-portable] edwin@k7>makeb le] edwin@k7>make install
a re up to date. If you are absolutely sure you want to override this
./bf-test.out &
../config.h ../config.h
:-) but if people are talking about HP-UX, Cray, ILP64 and epcdic2ascii(), I know it's either too difficult for me (You are not supposed to understand this) or it's bullshit (We can charge the phaser-array via a shortwave link through the warpcore). Time to startup vmware and run the experiment: gcc -o bf-test bf-test.c.
:-)
:-). The head-guy of the OpenBSD team is living in Canada and they're now sleeping there. I've spend a couple of days on #freebsd on irc.openprojects.net, so I just tried #openbsd.
:-)
01 August 2002 - 19:10:23 - OpenSSH 3.4p1 package trojaned
And all I was thinking was "Oh! I should upgrade ssh on these two machines before there are problems...". The beauty of FreeBSD is that it goes like this:
[~] edwin@k7>cd
[/usr/ports
[/usr/ports/security/openssh-porta
Easy euh? It went well, except for the second step:
===> Extracting for openssh-portable-3.4p1_7
>> Checksum mismatch for openssh-3.4p1.tar.gz.
Make sure the Makefile and distinfo file (/usr/ports/security/openssh-portable/distinfo)
check, type "make NO_CHECKSUM=yes [other args]".
*** Error code 1
Euh... I didn't remember seeing a change in the FreeBSD ports regarding this. And I didn't see an announcement for it from the people from OpenSSH... Oh well, it happens. I downloaded the new openssh-tarball:
-r--r--r-- 1 12187 mirror 840574 Jul 31 16:47 openssh-3.4p1.tar.gz
-r--r--r-- 1 12187 mirror 232 Jun 26 08:20 openssh-3.4p1.tar.gz.sig
That's weird, they've rerolled the tarball without updating the signature file. I asked a couple of people on irc (#sage-au) if they have had troubles with compiling openssh the last days. Yups, ^Sarge^@bofh.snsonline.net also had it, he had a checksum mismatch.
Curious as I was, I extracted the old and new tarball and this were the differences:
[~/test] edwin@k7>diff -r -u openssh-3.4p1-old openssh-3.4p1
diff -r -u openssh-3.4p1-old/openbsd-compat/Makefile.in openssh-3.4p1/openbsd-compat/Makefile.in
--- openssh-3.4p1-old/openbsd-compat/Makefile.in Wed Feb 20 07:27:57 2002
+++ openssh-3.4p1/openbsd-compat/Makefile.in Thu Feb 1 08:52:03 2001
@@ -26,6 +26,7 @@
$(CC) $(CFLAGS) $(CPPFLAGS) -c $bf-test.out; sh
$(COMPAT):
$(OPENBSD):
Only in openssh-3.4p1/openbsd-compat: bf-test.c
At this moment I asked a couple of people on irc (#sage-au) if they have had troubles with compiling openssh the last days. Yups, ^Sarge^@bofh.snsonline.net also had it, also a checksum mismatch. Time to go deeper into it...
bf-test.c is a weird file. It talks about HP-UX PL.2 systems, it talks about _CRAY notes, it talkes about none-T3E machines, it talks about _ILP64__ and it does an epcdic2ascii() call. I'm not very skilled in computers (well, I am
bf-test itself is pretty harmless, it only prints things to the screen (remember the change in the makefile? execute, redirect the output and execute the output). The shell script it prints creates a C program and tries to compile it. If it doesn't succeed at first, it tries to link other libraries (everybody who has ever ported a Solaris knows that you have to explicitely link to libresolv et al). So it's cross-platform
The C code is not that smart. It tries once per hour to connect to port 6667 on the machine 203.62.158.32 which is web.snsonline.net and waits for commands from the person or persons who 0wn3d the machine. Does it get an M, it sleeps for another hour. Does it get an A, it will abort. Does it get an M, it will spawn a shell. Some people will build it "normal" privileges and install it as root: they will get a shell with "normal" privileges. Other people will build it with "root" privileges and the shell will have "root" privileges.
While analyzing the code on #sage-au and mentioning the hostname, ^Sarge^ looked strangely at me (well, it's IRC so you never know but that's what I would do): "That is my machine.". The good news is that I didn't have to worry about finding out who manages the machine!
The next step is to inform somebody who manages the openssh-packages: The OpenBSD team. Up to right now, I have had no experience with the OpenBSD team (if you check my website you'll see that I'm more a FreeBSD guy
*** MavEtJu has joined #openbsd
Euh... anybody from the openssh-team here?
I have some news for you...
What's up?
I have contact! Marius asked me the standard questions (how did you find out, how can I see it, when did you find out) and after some investigation he said "I think I'd better call (and now I have forgotten the name)". Coolies! I think I found a right person to talk to! It looks like things are going to roll now, I can take my hands of it.
The last things I did were writing some emails to a couple of mailinglists and guide ^Sarge^ to #openbsd. For the rest I wasn't of very much use anymore, so I just kept monitoring #openbsd. And the logfile of my website, which went ballistic.
Aftermatch
* The portable version wasn't the only which was trojaned, the normal version was also.
* It seems it took only six hours before somebody was alert enough to see that there was something wrong, all thanks to the checking of the MD5-checksum [insert a sweet 'aaaaaahhh' here]
* OpenSSH itself wasn't trojaned, the tarball was. There is nothing wrong with OpenSSH itself (this time
* The building of a port (under FreeBSD at least) is done as root with all its privileges. This is a wrong approach. For a time I tried, as an experiment, to build ports as user "port". This worked fine except for the "make clean" part, in which I couldn't remove the files created during the "make install" phase and the files which were made during the building of the RUN_DEPENDS ports.
bash$
The machine was rebuilt from source and rebooted within an hour of finding out. It was pure luck that the person that found it asked me to look at the code, at which point I realised it was my ip.
Cheers,
^Sarge^
...for hosting ftp.openbsd.org on a box running SunOS, not OpenBSD!
It was "no remote holes in 5 years". Now it's "one remote hole in the default install, in nearly 6 years!"
Next it will be "one remote hole and one 'harmless trojan' in the default install, in really very close to 6 years!"
I read it first on gentoo-dev. The ebuild is not affected. The checksum in the ebuild will fail against the compromised tarball.
If there would be some configure/make environment that prevents or asks before outgoing connections and checks for possibly dangerous commands, that are unusual to call upon a ./configure run, wouldn't that prevent things like this to happen again?
Yes, I recommend having the installation banned from creating / deleting / running any files.
It seems like we need to start using a "web-of-trust" based PKI solution, like OpenPGP. And educating users to actually check the signatures!!!
On a related note, does Debian use anything to prevent this from happening? I for one don't worry too much when doing an update, maybe I should...
--
Adam Sherman
Freelance Geek
This is a solved problem. Red Hat, for example, GPG signs the MD5SUM file. So you can verify that the person who created the MD5SUM was authorized to do so.
This was just one type of trojan. Some others could go dormant for several hours before contacting the world outside. Simply "building the binaries with plug off the wall" is not a solution. It's a knee-jerk reaction and still at fault. The correct way is to check the package against a MD5sum or (preferably) GPG signature - and if possible, these should be at a different machine on a different network from the tarballs.
On the other hand, just looking at the trojan source quickly, it looks very much like a slightly evolved version of those found in irssi, BitchX, dsniff and fragroute configure scripts. This has already been noted by some other individual here as well. See his post for links.
There is no such thing as good luck. There is only misfortune and its occasional absence.
Check out this little snippet (the whole message can be found on lwn.net) from an email from Theo:
We've been trying to warn vendors about 3.3 and the need for privsep, but they really have not heeded our call for assistance. They have basically ignored us. Some, like Alan Cox, even went further stating that privsep was not being worked on because "Nobody provided any info which proves the problem, and many people dont trust you theo" and suggested I "might be feeding everyone a trojan" (I think I'll publish that letter -- it is just so funny).
Please do publish that letter, Theo. That would be very interesting.
PU
--Lawrence Lessig for Congress!
I'm one of the admins for SunSITE Alberta which houses openbsd.org. I just checked the file currently available for download and it seems to be clean. The MD5SUM matches up, as well as extracting and looking at the source bf-test wasn't present.
This really sucks since I woke up only like 10 minutes ago and find that the most downloaded file from your site may be trojaned. I have a distinct feeling that the rest of my day isn't going to be much better.
But in this circumstance, I don't believe it is the case. The trojan connects to port 6667, which is usually ircd. Outgoing connections to irc servers are not exactly uncommon in those boxes that run any kind of flavour of *nix. Hence, it's not a connection that really attracts attention by itself. It looks like a connection to a stand-alone ircd in netstat reading. Also, because irc is so common service to use, the firewall setups are likely to allow this through.
The other end of that connection, however, was more than likely running something totally unrelated to irc. After all, the connection itself is somewhat like a backward rsh. (I believe it actually bears the name "bindshell"...) This was a very basic case of trojan: install a backdoor that calls home and allows to execute commands remotely.
There is no such thing as good luck. There is only misfortune and its occasional absence.
Actually, they aren't the ones running the box. openbsd.org and openssh.org (including the main ftp servers) are run on Solaris at the University of Alberta in Calgary. This is because the Universtity has offered free bandwidth, and for projects as large as openbsd/openssh, free bandwidth is a godsend.
Since the trojan dies if it sees an A first thing, obviously the guy running the box it's trying to contact should run something like this:
yes "A" | nc -p 6667
Then every daemon that connects gets an A right away, and thus dies. End of problem.
At least mafia-owned pizzarias make excellent pizza. Compare to Bill Gates.
End result: no one in Gentoo has been able to compile/emerge openssh for the last few days.
Which is good :-)
In particular, if the machine in question is a server (usually the reason you have SSH on a box), you should make every possible effort to remove outgoing traffic. There's usually no reason for a server to create outgoing connections to the internet, and if it needs to connect to any specific local resources (e.g. a database machine), limit the iptables/routers appropriately.
Actually, that's not true. (someone correct me if I'm wrong on this, but I think I'm close). When you compile gcc, it first compiles a version of itself (stage 1) using your current gcc. Then it uses stage 1 to compile itself again, resulting in stage 2. Finally, it compiles itself one more time using stage 2, resulting in stage 3. The build succeeds if and only if stage 2 and stage 3 are identical. This process ensures that your final gcc is free from any defects in the original gcc. Once you have a clean gcc, you can rebuild the rest of your system using it.
Take a look at www.linuxfromscratch.org to see how this works for an entire system.
Step 1: Read Ken Thompson's Turing Award lecture "On Trusting Trust"
http://www.acm.org/classics/sep95/
Step 2: Decide for yourself if you're ready for the tinfoil-helmet brigade.
Step 3: Type 'make world' if you dare.
Recompiling the compiler doesn't do anything to rid you of trojan code/back doors. If you need to ask why, take a look at "Reflections on Trusting Trust" by Ken Thompson, and the description of a back door in the jargon file.
The short story is, that the compiler decides what to output. So you have to trust your compiler.
The reason for the recompiling of gcc is something else entirely: It is to allow the source code of gcc to rely on gcc, and to allow an optimized compiler to be created.
For additional information consult a good book on compiler writing.
Actually, the code wasn't "signed" in the cryptographic sense. The code was checksummed, and that checksum showed there was a problem. However, there is nothing stopping someone from modifying the checksum and making the archive appear legit.
Real cryptographic signing, like that mentioned in the grandparent post, involves hashing the tarball using a strong hash, and then encrypting that hash using the private key of the signer. Then, any person can retrieve the public key of the signer and verify that the "signature" is legitimate by attempting to decrypt the hash (this is probably not strictly correct with the way things really work, but the concept is, AFAIK, correct). The point is that the only person who can "sign" the tarball is limited to only legitimate people (or organizations). So, if the signature for the tarball is valid, you can guarantee it was signed by someone you trust, and so you can trust the code.
Alan Cox was calling Theo to task because he didn't like how Theo concealed the exact security problem until a workaround was given out. This is an attitude some developers have. It's not the best attitue from a customer/end-user standpoint, but some people who write code and give it for free use still don't understand it. Alanx Cox sounds like, despite him being a valuable asset to the community, he does not understand this.
If he'd have said, "for all we know, OpenBSD could attract near-earth bodies" would you post this comment as "eerily prescient" on the recent asteroid stories? Sometimes things just aren't related. Despite what Mulder may think.
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
why now? this whole episode seems to be a good example of the current system working well... tarball trojaned, ports system detects md5 mismatch, no compromise, no problem.
sic transit gloria mundi
And it looks like they're not "eating their own dog food," and eating Sun dog food instead
did you ever think there might a reason for that?
then you can't trust a web server to give you a web page with an unaltered MD5 sum. Surely this is common sense?
I am not sure, but this just might be the reason why systems like BSD ports and Gentoo portage store the MD5 sums in the ports trees, and don't in fact get them from websites.
The real solution is digital signatures (i.e. an MD5 sum encrypted with a private key).
WOW! what an original and fresh solution! you sir, are some sort of genious for coming up with this.
congratulations, you've managed to regurgitate several of the things that have been said, literally, hundreds of times today already. I think the Society for Prevention of Cruelty to Dead Horses might have a bone to pick with you.
sic transit gloria mundi