OpenSSH Package Trojaned

cperciva writes "The original story is here. And more details are available from the guy's weblog here." Here's a mirror of that email message. Another reader writes, "Not really a trojan because all it does is make a connection to 203.62.158.32:6667." Still another writes "The tarball of the portable OpenSSH on ftp.openbsd.org is trojaned. The backdoor is only used during build - generated binaries are fine." There isn't much authoritative information available, but this appears legitimate - please be careful if you're updating any of your machines with code from ftp.openbsd.org, and we'll update this story with more links as information is available. Update: 08/01 19:13 GMT by M : OpenSSH now has an advisory.

Comments

Since its only a build issue...

[LT4Ryan]

Why not unplug your box from the network while building? After that it should be OK, seeing as how 'generated binaries are fine.'??

Or am I thinking far too simply for my own good again? :)

Security, Antisecurity and a Purposeless Anecdote

[f00Dave]

On the one hand, there's stories about the improved security and paranoia of OpenSSH.

And on the other hand, there's stories like this one and that one about anti-security "features" in the same package.

Now, my question is this: is this indicative of open-source development projects, in general? [Yeah, it's faster to fix issues, but if the distros are *causing* issues in the first place, well.... ;-) ]

Reminds me of a company I worked for that was timebombed by a previous programmer. Unfortunately for him, when we looked at the source code, all was well (he'd copied the sources back over his modified ones used in the binary build) ... but he'd left the .bak files. Guess what was in the .bak files. Good, now guess how we discovered a few other potential surprises he'd left for the rest of us to encounter.

Anyway, I can't see how a disgruntled coder could really affect an open-source project, unless there's personality factors at play that I don't know about. Anyone have some meat on this OpenSSL mess?

Checksum...?

[DJPenguin]

OK so they trojaned the source tar.gz, and uploaded it to the server somehow. So why did they not update the MD5SUM also?

Trojan

[GigsVT]

The C code is not that smart. It tries once per hour to connect to port 6667 on the machine 203.62.158.32 which is web.snsonline.net and waits for commands from the person or persons who 0wn3d the machine. Does it get an M, it sleeps for another hour. Does it get an A, it will abort. Does it get an M, it will spawn a shell. Some people will build it "normal" privileges and install it as root: they will get a shell with "normal" privileges. Other people will build it with "root" privileges and the shell will have "root" privileges.

Tell me how this isn't a trojan again? A remotely controllable program that could possibly give the attacker root access?

Re:I'm suprised...

[Queuetue]

This shows why I trust OS peer-reviewed code... It only takes one curious person to find an exploit, and OSS allows that person to be anyone. This one was found in 6 hours, by someone who wasn't on the OpenBSD team or the OpenSSH team.

It's also why I spend (some say waste) a few idle cycles now and again just perusing code - it only takes one person to notice an anomaly. The more aggregate cycles spent reviewing code, the better the systems get.

Trojaned source distributions

[dzym]

So far we've seen dsniff and other programs from monkey.org trojaned, irssi, BitchX, and now OpenSSH.

At this point I think we need to make the assumption that the problem is a bit more common than viewing these compromises individually would suggest, and perhaps these individual events can even be linked together.

And for the developers out there, I think it's time to check over all of your current distributed source tarballs.

Re:Trojaned source distributions

[Anonymous Coward]

Okay, publicly available security-related software
is and always has been vulnerable to hacking - no more (and probably less than) your average commercial product.
What is different about open source security software? It is uncontrollable - it is out there, replicated, digested and understood by many people.
So, take on the role of someone wanting you to stop using open source security software, what do you do?
You can't directly supress (ban) the products without showing your hand, so, as far as I know, the only thing you can do is undermine the respect for the product.
Slashdot readers will happily recognise moves by Microsoft to stiffle and control, but what would should you look for when talking about security software?
Who are the players? Who are the vested interests? How can the manifest their influence without showing their hand?

OpenSSL, OpenPGP et al *ARE* a thorn in the side of some organisation's agendas. If you think otherwise, you simply aren't thinking. So what do they do?
Undermine. Now, I'm not saying that THIS specific is a case of organised disinformation, but simply, well, HEADS UP!

looks like it's from our australian friends

[Anonymous Coward]

inetnum 203.62.158.0 - 203.62.159.255
netname AUSTRALIANINTER-AU
descr Australian Internet Solutions Pty Ltd
descr Suite 3, Level 5, 277 Flinders Lane
descr Melbourne
descr VIC 3001
country AU
admin-c DA53-AP
tech-c DA53-AP
mnt-by MAINT-AU-KALED
changed register@aunic.net 19970211
changed aunic-transfer@apnic.net 20010525
changed hostmaster@apnic.net 20011115
source APNIC

person Domain Administrator
address Level 4,
address 180 Bourke St,
address Melbourne, 3000.
country AU
phone +61-3-9650-5566
fax-no +61-3-9639-1897
e-mail kaled@dalek.ains.net.au
nic-hdl DA53-AP
mnt-by MAINT-NEW
changed kaled@dalek.ains.net.au 20010619
source APNIC

Re:How many people do check the MD5 checksum?

[stevey]

Do you check the packages downloaded from sites that you usually do not have problems with?

I've been wondering about this - and the answer is almost certainly not.

I've written a fairly widespread mp3/ogg streamer. I used to list MD5 sums on the download page - but recently I've switched to signing with my GPG key.

(On the basis that if somebody altered the downloads they'd be capable of fixing up the MD5sums file in the same directory too).

Taking a look at the download statistics you can see that about 1 person in 50 downloaded the signature file to match their archive.

That suggests that 2% of people routinely check signatures. I assume that less people check the code than check the signatures so ... it's probably safe to say that no more than 0.5% of people do.

Re:How many people do check the MD5 checksum?

[zmooc]

What we need is a trusted 3rd party that has all the checksums. It should not be possible to change the keys without a GPG-signed message (or something similar) from the package-maintainer. Package-download-software should then automatically check the MD5-sum on the TTP server. Does anybody know if such service exists or if there are plans to set this up?

Re:How many people do check the MD5 checksum?

[Quixote]

Thats what I was thinking, too.
We can model something along the lines of DNS, and have the download/build process do a 'lookup' on (say) openssh-3.4p1.packages.net, to get the MD5 sum, and compare it with whats on hand.

Never underestimate the power of a bunch of pissed-off nerds... :)

Re:203.62.158.32

[Anonymous Coward]

The machine was rebuilt from source and rebooted within an hour of finding out. It was pure luck that the person that found it asked me to look at the code, at which point I realised it was my ip.

Cheers,

^Sarge^

Re:what's up with OpenBSD?

[Mercaptan]

It's not in any trouble at all.

OpenBSD is less of a fortress and more of a flexible defense. In this case, even though the integrity of the centralized source code was compromised, any end-user who accessed it via the ports tree was immediately tipped off that something wasn't kosher. They could then communicate this to other users and the maintainers of OpenBSD and thus make this attack known to the public within hours of it happening. And due to the ease of updating that the ports tree provides, the maintainers of OpenBSD can correct this problem very quickly. This sort of suppleness provides for the best kind of broadband defense, whereas a "fortress" cannot brook much weakness in any of its parts and is far more brittle. Had users not been able to see the disparity (via MD5 sums), or not been able to communicate it to their fellow users, or not have been able to easily obtain a clean copy, then the problem may have been easily transmitted to a large number of operating OpenBSD machines. As it was, the problem got nipped at the bud.

This event would be the sort of reason why security-conscious people should stick with OpenBSD.

The real issue is persistence

[f00zbll]

Other's haven't mentioned this, but I think the real issue here is persistence. If I buy a piece of software from company A, I can't just sit around and read the source code to figure why it works a certain way to figure how to best use the application. This means the only eye balls reading the code for fun or for real reasons are new programmers joining company A. How likely is a closed environment going to encourage programmers and user to explore and look at the code? Trojans and virii will happen, it's a given. Encouraging people to look at the code is your best bet.

How many companies are going to tell a new programmer to go ahead and spend 6 months reading through all the code? How many companies encourage all the programmers to look at old code, check every line every couple weeks and perform extensive regression testing? From my own experience, few companies look at old code and the regression testing is typically a narrow focus on the functional aspects. Things like a trojan aren't going to be caught by the typical regression testing procedures.

On my free time, I do read through open source code for fun. From my own biased experience, open source code tends to be much cleaner and better documented than closed source projects. This isn't including all the PERL code I've seen written in creative ways to make visual art. I've also seen clean PERL code, but that's another story. Back to the point. Persistence is what wins in the end when it comes to security. The minute a person get lazy is when an attack will happen. But I seriously doubt this will change in the near future, since it's really a matter of culture. Businesses can't afford to have a team of programmers to sit around and audit their security every couple months. So unless our culture changes and realizes quality is more important than convienance, things like this will continue to increase in frequency. Of course everyone living in a modern techno society is guilty of it. But that's not to say technology is the cause of it, though they are related.

Open Source PKI Needed?

[carbon60]

It seems like we need to start using a "web-of-trust" based PKI solution, like OpenPGP. And educating users to actually check the signatures!!!

On a related note, does Debian use anything to prevent this from happening? I for one don't worry too much when doing an update, maybe I should...

hostinfo

[suwain_2]

$ hostinfo -n 203.62.158.32
snsonline.net
$ whois 203.62.158.32

% How to use the APNIC Whois Database www.apnic.net/db/
% Upgrade to Whois v3 on 20 August 2002 www.apnic.net/whois-v3
% Whois data copyright terms www.apnic.net/db/dbcopyright.html

inetnum: 203.62.158.0 - 203.62.159.255
netname: AUSTRALIANINTER-AU
descr: Australian Internet Solutions Pty Ltd
descr: Suite 3, Level 5, 277 Flinders Lane
descr: Melbourne
descr: VIC 3001
country: AU
admin-c: DA53-AP
tech-c: DA53-AP
mnt-by: MAINT-AU-KALED
changed: register@aunic.net 19970211
changed: aunic-transfer@apnic.net 20010525
changed: hostmaster@apnic.net 20011115
source: APNIC

person: Domain Administrator
address: Level 4,
address: 180 Bourke St,
address: Melbourne, 3000.
country: AU
phone: +61-3-9650-5566
fax-no: +61-3-9639-1897
e-mail: kaled@dalek.ains.net.au
nic-hdl: DA53-AP
mnt-by: MAINT-NEW
changed: kaled@dalek.ains.net.au 20010619
source: APNIC

I've apparently triggered the lameness filter with this... BTW, I can ping this host, so it's still up. However:

$ telnet 203.62.158.32 6667
Trying 203.62.158.32...
telnet: Unable to connect to remote host: Connection refused

Looks like they closed that port?

Re:How to stop this happening again?

[Hanashi]

I think a multi-pronged approach is in order.

First, open source project teams should try their best to make sure that their distribution points are secure. I know that no one intentionally distributes code from known-bad servers, but rather than just assuming security in the absence of knowledge to the contrary, I think they should make the security of their servers more of an active part of their project.

Second, they should register PGP keys with the key servers. Popular projects could even sign each other's keys, creating an informal web of trust. For example, the FreeBSD, OpenSSL and OpenSSH teams could maybe sign each others keys. This would help users make sure they don't download false keys from the servers.

Project teams should keep their private keys on read-only media (like a CD-R), and only 2 or 3 trusted members should be allowed to sign distributions. Really large projects could even do key splitting, so that (for example) 3 out of 5 developers are required in order to sign a release.

Of course, the project files should have detached signatures available in the same place that the distribution is. If it's FTP, put the detached signature in the same directory as the tarball. A lot of projects already do this. A pointer to the signature should be included in the project. Preferably, this should be in a standard file, like the $PROJECTROOT/SIGNATURE file, to make it easy to find.

If all these suggestions (or something like them) were followed, then someone who downloaded a package could use an automated tool to check the signature in one easy step. It is almost axiomatic that convenient security procedures get followed far more often than inconvenient ones, so I think automation is an important consideration here. It would add only one easy step, transmuting

configure && make && make install

to

checksig && configure && make && install

Portscan of the backdoor IP

[CajunArson]

Here is an nmap dump of the IP in question that
the backdoor tries to connect to:

nmap options (where options is filtered by Slashdot)

ALRIGHT FSCK THIS!! You'll just have to take my
word for it the nmap showed the port closed (do it yourself) I've just tried 10 different ways to submit the nmap output and the lameness filters won't let it through.

Note that port 6667 does not appear to be open, although a backdoor is still a pretty big thing
to worry about. Also note that much of the output
is cut out due to LAME Slashdot filters.

Re:Why not hack the md5 checksums?

[disappear]

So, in this case, couldn't someone just as easily generate an md5 sum for the hacked file and put that in the sum file? I know on bsd you have ports which would prevent this, but what about Linux?

This is a solved problem. Red Hat, for example, GPG signs the MD5SUM file. So you can verify that the person who created the MD5SUM was authorized to do so.

Prescient Alan Cox / Theo exchange

[wfrp01]

Check out this little snippet (the whole message can be found on lwn.net) from an email from Theo:

We've been trying to warn vendors about 3.3 and the need for privsep, but they really have not heeded our call for assistance. They have basically ignored us. Some, like Alan Cox, even went further stating that privsep was not being worked on because "Nobody provided any info which proves the problem, and many people dont trust you theo" and suggested I "might be feeding everyone a trojan" (I think I'll publish that letter -- it is just so funny).

Please do publish that letter, Theo. That would be very interesting.

PU

suspicious! person who reported KNEW ip owner

[Anonymous Coward]

If you look at the reporting site, the person who reported it alleges that he went into an IRC channel, and immediately ended up speaking to the very owner of the machine which the trojan connects to.

Now, what are the chances of that happening? Finding a random trojan on your machine, popping in to an IRC channel, and immediately walking in on the owner of the machine from which the backdoor seems to originate? Who then immediately states "ah yes, that's my machine!" and closes the offending port. Well, how many machines are there on the 'net -- you work it out.

Sorry, my friends, but this smells of bullshit to me. I think the guy who reported it knows a lot more about it than he's making on... if he didn't write and install the bf-test himself, I would hazard a guess he knows who did. And, as the guy who reported the issue, he is least likely to be suspected... which makes the whole thing all the more cunning, if my suspicions are correct.

Could Freenet, GnuNET help here?

[grumbel]

As it seems to happen more and more often that ftp servers get cracked and md5sum's don't seem to help (since users are to lazy to check them and the gpg signatures), could peer2peer services provide a solution? With things like GnuNET you don't download an URL, but instead a checksum. So there wouldn't be an easy way to replace a file in a single location, but one would need to spread a fake md5sum and make people belive that the fake one is the real one. With peer2peer systems there wouldn't be a single point of failure, where the file could get trojaned, once the correct md5sum is spread in the public it would be nearly unchangable. It would also be impossible to hijack mirrors or that trojaned files would be mirrored, since mirroring would be handled by the network itself, not by people. Well, its just an idea, but GnuNET and Co. seem to have a much more straight forward way of handling checksums, than you can ever get with http or ftp at the moment.

I think it's okay now

[hardave]

I'm one of the admins for SunSITE Alberta which houses openbsd.org. I just checked the file currently available for download and it seems to be clean. The MD5SUM matches up, as well as extracting and looking at the source bf-test wasn't present.

This really sucks since I woke up only like 10 minutes ago and find that the most downloaded file from your site may be trojaned. I have a distinct feeling that the rest of my day isn't going to be much better.

FYI: Gentoo OK

[jehreg]

Gentoo is a source-only distribution. This trojan has not affected Gentoo since the MD5 digest is checked before compilation occurs. I just checked, the MD5 digest included in the "portage tree" is the correct one, and portage has detected the change.

End result: no one in Gentoo has been able to compile/emerge openssh for the last few days.

Which is good :-)

Interesting

[bythescruff]

"At this moment I asked a couple of people on irc (#sage-au) if they have had troubles with compiling openssh the last days. Yups, ^Sarge^@bofh.snsonline.net also had it, also a checksum mismatch."

The trojan connects to 203.62.158.32, eh?

> nslookup 203.62.158.32

Server: dns2.intra
Address: 10.16.59.15

Name: snsonline.net
Address: 203.62.158.32

>

Seems like OpenSSH 3.4 trojaned too

[unixmaster]

From freebsd-security mailing list. I am not sure this is for real or fake

I just upgraded my OpenBSD 3.0 machine to OpenSSH 3.4 last night.
I downloaded openssh-3.4.tgz ( notice not p1 ). The MD5 I got was
MD5 (openssh-3.4.tgz) = bda7c80825d9d9f35f17046ed90e1b0a
And look :
[root@superfrink /root/upgrades]# tar -tzf openssh-3.4.tgz | grep bf
ssh/ssh-keygen/bf-test.c
And then:
[root@superfrink /root/upgrades]# head -5
ssh/ssh-keygen/bf-test.c
/* * Blowfish input vectors are handled incorrectly on HP-UX PL.2 systems.
* Perform routine compatability checks. */
#include
So I guess It's not just openssh-3.4p1.tar.gz that is trojaned.
/Chad