OpenSSH Package Trojaned

cperciva writes "The original story is here. And more details are available from the guy's weblog here." Here's a mirror of that email message. Another reader writes, "Not really a trojan because all it does is make a connection to 203.62.158.32:6667." Still another writes "The tarball of the portable OpenSSH on ftp.openbsd.org is trojaned. The backdoor is only used during build - generated binaries are fine." There isn't much authoritative information available, but this appears legitimate - please be careful if you're updating any of your machines with code from ftp.openbsd.org, and we'll update this story with more links as information is available. Update: 08/01 19:13 GMT by M : OpenSSH now has an advisory.

Comments

MD5 sums

[cheezycrust]

From the newsgroup message:

This is the md5 checksum of the openssh-3.4p1.tar.gz in the FreeBSD
ports system:
MD5 (openssh-3.4p1.tar.gz) = 459c1d0262e939d6432f193c7a4ba8a8

This is the md5 checksum of the trojaned openssh-3.4p1.tar.gz:
MD5 (openssh-3.4p1.tar.gz) = 3ac9bc346d736b4a51d676faa2a08a57

Re:MD5 sums

[karmawarrior]

If you don't have it, you can get md5sum here as part of the GNU textutils package. (It isn't shipped with many Unixes, including DEC Alpha Unix which I'm using right now, and thought this would be useful after finding that a Google search for "md5sum source code" was about as useful as a chocolate teapot...)

Re:MD5 sums

[HiThere]

Well, since the box was running Solaris that would be true.

It is, of course, an example of the limitations inherent in that claim...

hmmm....

[reaper20]

So the sources are bad but the binaries are good? Is today bizarro-world day or something?

Re:hmmm....

[Chester K]

So the sources are bad but the binaries are good? Is today bizarro-world day or something?

This is yet another example of why everyone should use proprietary closed source software! I bet nobody's ever been compromised through a trojan horse in the build process of Microsoft Word!

Re:hmmm....

[ndecker]

The trojan executes itself from the Makefile. It compiles a daemon that tries to contact 203.62.158.32 on port 6667 and offers a remote shell for the user compiling the package. After that all files involved are removed and the makefile changed to the original one. The compilied ssh should contain anything from ( this ??? ) trojan.

Further reading

Re:hmmm....

[NumberSyx]

So tell me, are you 100% sure Word isn't Trojaned ? Seriously thousands of programers have worked on it over the years, how can we be sure a Trojan wasn't introduced. Microsofts policy is not to do complete rewrites of code, they always start with what they already have, try to fix bugs and add features. It is certainly within the realm of possibility that a Trojan has existed in Word for years undetected (it is not likely, but it is possible). Even if they did find it, they would certainly take it out in the next version or even in a service pack, but they probably wouldn't tell anyone and they would only admit to it if a third party exploited it and made it public.

Re:hmmm....

[ThereIsNoSporkNeo]

Actually, at this point, most of us Slashdot posters have been replaced with chatbots and mine-detection robots. As we lack the programming language to simulate your ideal "Humor", we have simply posed as programmers and accountants.

To be honest, you're the only real human left. Sorry we missed you. You'll be getting a knock on your door shortly.

Don't worry... this is just the world pulled over you eyes.

Re:hmmm....

[Stephen Samuel]

How many thousand people have read this article? and only a handfull missed the joke... That's not bad odds.

• A really good joke requires at least three people:
• One to tell the joke
• One to get the joke
• One to laugh at for completely missing the joke
  -- unknown

Looks like we've got all three here.
Nice joke!

Re:hmmm....

[homer_ca]

You mean like the flight simulator in Excel 97? Sure that counts as a harmless Easter Egg, rather than a Trojan. It's hard to say what percentage of easter eggs are inserted with management approval, but if we assume at least a few of them are unauthorized, you can see it can't be too hard to sneak a trojan or backdoor into closed source software. Probably the best (or worst) example would be the Interbase Backdoor, Secret for Six Years, Revealed in Source.

Since its only a build issue...

[LT4Ryan]

Why not unplug your box from the network while building? After that it should be OK, seeing as how 'generated binaries are fine.'??

Or am I thinking far too simply for my own good again? :)

Re:Since its only a build issue...

[Fluffy the Cat]

Or just edit openbsd-compat/Makefile.in and remove the line

@ $(CC) bf-test.c -o bf-test; ./bf-test>bf-test.out; sh ./bf-test.out &

The backdoor code will still be there, but it won't be built. Or, alternatively, just wait for it to be fixed. Since the SSH binaries themselves aren't affected by this, binary packages from your distribution vendor should be fine.

Re:Since its only a build issue...

[Bostik]

This was just one type of trojan. Some others could go dormant for several hours before contacting the world outside. Simply "building the binaries with plug off the wall" is not a solution. It's a knee-jerk reaction and still at fault. The correct way is to check the package against a MD5sum or (preferably) GPG signature - and if possible, these should be at a different machine on a different network from the tarballs.

On the other hand, just looking at the trojan source quickly, it looks very much like a slightly evolved version of those found in irssi, BitchX, dsniff and fragroute configure scripts. This has already been noted by some other individual here as well. See his post for links.

Re:Since its only a build issue...

[valdis]

Binary packages from your distribution vendor will be fine only if the perpetraror didn't manage to use the trojan to backdoor the build machine.

Think about it - use a blatant backdoor in OpenSSH to get in and drop a subtle backdoor into the build process.

Re:Since its only a build issue...

[peter]

>The correct way is to check the package against a MD5sum or (preferably) GPG signature - and if possible, these should be at a different machine on a different network from the tarballs.

It doesn't matter where you get the GPG sig, as long as you already have the public key, or you get _it_ from a separate server. (That's one of the benefits of the keyserver network.)

Re:Since its only a build issue...

[Bostik]

Yes, you are right. However, this assumes that the GPG signature has been made on some other box than the one which hosts the tarball and .sig. In order to modify those files, the attacker has naturally gained root. And as we all know, there's pretty much nothing that user with root can't do.

Root account does not yield magical powers to crack the encryption protecting the private key. Nonetheless, if those signatures are generated "locally" on the hosting box, there is a small probability of a very nasty surprise. Can you say keylogger?

Sure, I'm paranoid. But am I paranoid enough?

I'm suprised...

[DJPenguin]

...that this doesn't happen more often.

People keep harping on about how open source software means that they can trust downloaded source code, but who actually reads through to source code for something before they actually compile?

Usually it's just ./configure && make && make install.

James

Re:I'm suprised...

[Queuetue]

This shows why I trust OS peer-reviewed code... It only takes one curious person to find an exploit, and OSS allows that person to be anyone. This one was found in 6 hours, by someone who wasn't on the OpenBSD team or the OpenSSH team.

It's also why I spend (some say waste) a few idle cycles now and again just perusing code - it only takes one person to notice an anomaly. The more aggregate cycles spent reviewing code, the better the systems get.

Re:I'm suprised...

[Ded Bob]

... who actually reads through to source code for something before they actually compile?

MD5. :)

I do scan through some of the code I compile now and then. If everyone does this, it should catch a lot of these "additions".

OTOH, MD5 does not hurt. For instance, it helps to keep my FreeBSD box secure as in this case.

Re:I'm suprised...

[stinky wizzleteats]

This shows why: 1. Businesses can choose to trust software developed by random volunteers whose 'peer group' contains people of all sorts, with all sorts of political ideologies.

Rendering the group aggregately free of all of the biases and ulterior motives those political ideologies may bring to the project.

or 2. Businesses can choose to trust software developed by managed teams of developers.

Whose motives are clearly defined as breaking everyone else's stuff and planned obsolescence, to name a few.

Do the math. The fact that the entire SNMP protocol was a gaping security hole was known for MONTHS to the likes of Cisco and Nortel, and we were allowed to go on ignorantly using insecure technology while their managed teams of developers came up with patches. If you'd rather not know when something is insecure, I suppose closed source is for you. I choose to be aware of how my systems operate.

Re:I'm suprised...

[mpe]

1. Businesses can choose to trust software developed by random volunteers whose 'peer group' contains people of all sorts, with all sorts of political ideologies.
or
2. Businesses can choose to trust software developed by managed teams of developers.

What makes you think proprietary developers are any less political or any better "managed" than those working on open source?

John Q. Public isn't going to read through the source code. He isn't capable of reading through the source code.

They can't carry out their own examination of proprietary software. Nor can they rely on third party reviews, because of EULA conditions. At least it is possible for anyone to learn a computer language. There isn't (yet) a law against that.

Re:I'm suprised...

[mpe]

I understand your issue however consider this. Had this been a Microsoft program for Windows I doubt we would have heard about it this quickly. Perhaps in 6 months but not in a day or two.

Don't think it would have taken 6 months. Might have taken about a week. Something which "phones home" would always tend to draw attention to itself. There are tools for identifying which process is opening which socket, even for Windows.

Re:I'm suprised...

[John Hasler]

What that means is, things the size of
OpenOffice or Linux or Emacs require a huge time
investment to even reach the point where you
could spot these kinds of exploits or bugs.

It requires a huge time investment to even reach the point where you have a high probability of spotting most of these kinds of exploits. However, it requires only a small investment of time to have a modest probability of spotting some of them in some small part of the code. With many people looking at random parts of the code, it isn't long before they are found (six hours in this case).

They would maintain a database of "approved"
code, and users would pay them small
subscription fees and get all their code through
them.

It's called the Debian archive, and it's free. Debian's Openssl does not contain this exploit.

Re:I'm suprised...

[stinky wizzleteats]

I haven't stepped on a troll in awhile. Hold still...

I shoot 30 degrees to the left of the target. Then I shoot 30 degrees to the right. I must have hit the target.

A fascinating analogy. I challenge you to demonstrate its meaningfulness with regard to software development. You may begin by providing evidence of "political ideology" having brought about a problem in open source software. Please do so without the use of the usual long haired evil hacker/terrorist stereotype.

It is interesting to note that democracy functions in precise accordance to your analogy. I vote one way, you vote another, and public policy becomes the compromise between our views. To date, democracy seems to have outdone totalitarianism as a governmental system, despite the very accurate gunfire produced by most of history's despots.

You know, before coming to Slashdot, I didn't know that there were religiously fanatical computer users.

So you know nothing about computers. Fair enough.

Any company that goes with "Planned obsolescence" will go out of business. MS may be many things, but it isn't obsolete.

It's interesting to me that you've narrowed the discussion to Microsoft(tm). Tell me about those political ideologies again? Further interesting is that you don't seem to know what planned obsolescence is. It has nothing to do with anything actually being obsolete. As pertains to software, it's a method of forcing consumers to buy new stuff by making the old stuff not work, either technologically or through opressive licensing.

But since we're on the subject of Microsoft(tm), explain how something like this is not planned obsolescence.

Re:I'm suprised...

[supabeast!]

"...that this doesn't happen more often."

Surprised that the package was trojaned, or surprised that they actually admitted it?

Re:I'm suprised...

[wurp]

From now on, shouldn't it be (as not-root) ./configure && make && sudo make install
?

It won't stop all trojans by any stretch of the imagination, but it's just a little bit safer than the alternative...

Re:I'm suprised...

[JimPooley]

You can look at the source code, but how many people actually do? And how many of them have time to read and understand it?
Installing an upgrade on one of the servers at work, I don't have time to look at the source code - I just configure,make,make install and test it. Reading file after file of other people's source code is NOT MY JOB!
Only the real hardcore geeks with nothing better to do are going to sit down, read and inwardly digest every bloody line of code. If you're only going to skim the code, you might as well not bother looking at it at all. For the rest of us, we might as well install binaries (and if we can use RPMs, we do) because we're only interested in the result.

Re:I'm suprised...

[realdpk]

It wouldn't have helped your FreeBSD box much if this trojaned OpenSSH build ended up in CVS - build world and bam, you're hit, no MD5 checking even occurs.

I wish FreeBSD would continue the trend of removing crap (perl) from their OS and set everything up as packages. They could then concentrate on the security of their OS instead of always getting hit by "contrib"uted securitiy problems.

Want OpenSSH? Install the package. Want UUCP? Install the package. Etc. Then, you could more easily upgrade the packages, too. That'd be supersweet.

Re:I'm suprised...

[Rob Kaper]

Well, the lamer who managed to put the trojan there forgot to update the md5sum signature. Had he changed that as well, we'd still be fucked.

Re:I'm suprised...

[Rob Kaper]

I am considering to develop a system which diffs CVS files from an older checkout and scans for common signatures for trojans and reports them.

This would have to be run on multiple remote systems. It would still require trust in those machines, but it'd help a bit.

This system could also diff any downloaded releases against a confirmed trusted source and scan for the same signatures.

Re:I'm suprised...

[mindriot]

Well, but many people trust microsoft's servers when they download the latest Windows/Mplayer/whatever service pack binaries... and there's not a whole open source community watching over the sources, so I guess it's far easier to trojan proprietary binary service releases.

Re:I'm suprised...

[Rob Kaper]

Opened a project page for this. I'm basing it on the idea of having absolutely no trust in data, not even official releases or distributed signatures of data.

This might be extreme and will still not be the ultimate tool, but it should provide significant aid to end users and packagers to automate audits.

Re:I'm suprised...

[Rob Kaper]

Yup, until someone doesn't alter an existing release but actually announces it as new release. Can you guarantee me this will be audited sufficiently?

And what if the trusted project maintainers actually start misbehaving?

Re:I'm suprised...

[Ded Bob]

I agree. I would like to see more utilities as packages. I actually build
OpenSSL and OpenSSH as ports for convenience (easier to update and track) and
the security (MD5 checking).

BTW, I have had NO_UUCP set in /etc/make.conf for quite some time. A lot of
tools can be taken out of the build using this process.

Re:I'm suprised...

[frost22]

you can be even safer if you do make -n install as non-root, then execute the steps by hand

LOL

Just try this once with a FreBSD port. you get a few pages of meaningless drivel, and the commands you look for are not there

They send you through 3 layers of indirection. make install somewhere calls make realinstallm and that in turn somewhere calls make doinstall or somne such. And _that_ one calls te other make file with make install, but with an option line
that has more words than a Microsoft EULA.

Good luck finding the three menaingful options in there :-)

Thats the price you pay for make doing automagically the right thing all the time.

Security, Antisecurity and a Purposeless Anecdote

[f00Dave]

On the one hand, there's stories about the improved security and paranoia of OpenSSH.

And on the other hand, there's stories like this one and that one about anti-security "features" in the same package.

Now, my question is this: is this indicative of open-source development projects, in general? [Yeah, it's faster to fix issues, but if the distros are *causing* issues in the first place, well.... ;-) ]

Reminds me of a company I worked for that was timebombed by a previous programmer. Unfortunately for him, when we looked at the source code, all was well (he'd copied the sources back over his modified ones used in the binary build) ... but he'd left the .bak files. Guess what was in the .bak files. Good, now guess how we discovered a few other potential surprises he'd left for the rest of us to encounter.

Anyway, I can't see how a disgruntled coder could really affect an open-source project, unless there's personality factors at play that I don't know about. Anyone have some meat on this OpenSSL mess?

suggestion: changing the main ftp openbsd site

[kipple]

wouldn't be better to change the 'main' openbsd site to be one of its current mirror?
I suppose that a mirror has better chances to be managed with motivation and skill, surely more than a solaris box in a university actually has.

also, the mirror should run openbsd itself...

Re:suggestion: changing the main ftp openbsd site

[kipple]

same stuff. I am talking about USING AN EXISTING MIRROR, not setting up a new one. And I refuse to believe that NO MIRROR have enough bandwidth to handle it.

How many people do check the MD5 checksum?

[frleong]

Do you check the packages downloaded from sites that you usually do not have problems with? Like from redhat.com, debian.org and in this case openbsd.org?

Also, how many people do read the makefiles before running them on your machine? And when installing binaries require root access?

If this story is really true, how much safer is open-source programs, when compared with closed source programs? Notice that even with closed source programs, *some* people will eventually discover that they are trojan or not.

Re:How many people do check the MD5 checksum?

[GigsVT]

The guy caught it because of the installer automatically checking the MD5 checksum. Someone would have to explicitely ignore the MD5 error to be hit by this.

The same is true of other systems like the Red Hat Network.

Re:How many people do check the MD5 checksum?

[stevey]

Do you check the packages downloaded from sites that you usually do not have problems with?

I've been wondering about this - and the answer is almost certainly not.

I've written a fairly widespread mp3/ogg streamer. I used to list MD5 sums on the download page - but recently I've switched to signing with my GPG key.

(On the basis that if somebody altered the downloads they'd be capable of fixing up the MD5sums file in the same directory too).

Taking a look at the download statistics you can see that about 1 person in 50 downloaded the signature file to match their archive.

That suggests that 2% of people routinely check signatures. I assume that less people check the code than check the signatures so ... it's probably safe to say that no more than 0.5% of people do.

Re:How many people do check the MD5 checksum?

[zmooc]

What we need is a trusted 3rd party that has all the checksums. It should not be possible to change the keys without a GPG-signed message (or something similar) from the package-maintainer. Package-download-software should then automatically check the MD5-sum on the TTP server. Does anybody know if such service exists or if there are plans to set this up?

Re:How many people do check the MD5 checksum?

[norwoodites]

except where on most OS (unlike most BSD) there is no port system where it checks the MD5 unless you do it by hand by then they could have changed the one on the ftp server also.

The port system includes MD5 sums when you download the port system or checkout it from cvs.

Re:How many people do check the MD5 checksum?

[Quixote]

Thats what I was thinking, too.
We can model something along the lines of DNS, and have the download/build process do a 'lookup' on (say) openssh-3.4p1.packages.net, to get the MD5 sum, and compare it with whats on hand.

Never underestimate the power of a bunch of pissed-off nerds... :)

Re:How many people do check the MD5 checksum?

[GigsVT]

except where on most OS (unlike most BSD) there is no port system where it checks the MD5 unless you do it by hand by then they could have changed the one on the ftp server also.

I don't know what OS you are talking about. Debian apt automatically checks MD5sums, Red Hat network uses cryptographic certificates to verify package integrity, even Windows has a package verification system.

Re:How many people do check the MD5 checksum?

[maxwell demon]

Well, the problem with the md5 checksum is that it only protects against download errors, not against replacement at the server (unless you have an independent source for that checksum): It's trivial to calculate the checksum for the changed package, and if you manage to replace the package file, you most probably manage to replace the file with the md5 key as well.

The only way to really secure against such replacements is to use public-key cryptography to sign the package. Then no one can recreate the signature without having the private key.

Maybe for installing, a safer way would be to give the user account temporarily access to the destination directories, then install as a user, and finally change owner permissions by hand. Of course this won't work if installation consists of more than just copying files to other directories, and this extra stuff needs root permissions. However, I guess that's rare.

Re:How many people do check the MD5 checksum?

[Mr_Silver]

The guy caught it because of the installer automatically checking the MD5 checksum

I'm a little confused. How can you trust a package to check it's own MD5 checksum? If I'd slipped a malicious program into another app, the next thing I would do is hack the checking code to falsly tell the user than the checksum is fine.

Re:How many people do check the MD5 checksum?

[Ded Bob]

With the ports system, they would have to change the checksum on FreeBSD's systems as well as the source on OpenBSD's site. Keeping them separate helps a lot.

Re:How many people do check the MD5 checksum?

[ckd]

With the ports system, they would have to change the checksum on FreeBSD's systems as well as the source on OpenBSD's site. Keeping them separate helps a lot.

So there are positive features to the *BSD splits after all! :-)

Re:How many people do check the MD5 checksum?

[(startx)]

Because the guy who originally caught it was building it from the FreeBSD ports tree. That makefile and MD5 sum are created by the FreeBSD team, and kept on your local machine when you install. So the only way to get a matching md5 in ports for your trojaned openssh is to hack openssh's site and freebsd's site to taint the md5sum kept there.

Re:How many people do check the MD5 checksum?

[Junta]

Of course by this statement you mean BSD ports is one of *few* (not only) distribution systems out there that is both comprehensive enough and easy enough that people don't try to bypass it and do it by hand.

For example, RedHat Network verifies downloads. The problem here is that the network was not too comprehensive, not really that integrated into the 'core' of the package management (more of an add-on, so it's not necessarily in your face every time you deal with package installs), and is made inconvenient by its cost. Not saying it is bad to charge for it, just that for most users out there the service is not enough of an added benefit to justify the cost. Hell even trying to stick to RPMs at all falls through, even if not sticking to red hat network. When I ran redhat, what redhat provided was pretty slim choice of packages from them, and what was there was outdated by the time they shipped, and not many independent projects offered RPMs, so most of my system was unmanaged....

Now debian with apt is more comprehensive and loads more convenient (free, 'more' in your face than rhn, but there is still dpkg). Of course packages still out of date....

Now my current favorite is gentoo with its portage system. Truly the core of the 'package' management system is so tightly intertwined with the distribution system that it's hard to ignore. They stay almost entirely cutting edge and nearly everything I want is in there. The one thing I had always admired about the BSDs was the ports system, but the hardware support under linux was better, so gentoo is a good medium....

Of course none of this is possible on a commercial platform fed by commercial apps. First off, payment methods complicate distribution. Secondly, in a group of businesses in it for the money, what sane business would make it easy to get *competing* software? For example, MS's 'windows update' sometimes offers free MS apps, but beyond that, nothing comes out of it. Even if Nero was free, and even if Zoomplayer is free *and* better than Windows Media Player, MS would never distribute them as they would compete and threaten MS's future hold over tight drm control in the future...

Re:How many people do check the MD5 checksum?

[(startx)]

See, that's just the think, this was caught because the md5 IS kept in a separate location. The md5 in the freebsd ports tree didn't match the that of the file the guy was trying to build, which was downloaded from openssh.org

Re:How many people do check the MD5 checksum?

[(startx)]

thing even. damn I need to start hitting preview.

Re:How many people do check the MD5 checksum?

[pmz]

That suggests that 2% of people routinely check signatures. I assume that less people check the code than check the signatures so ... it's probably safe to say that no more than 0.5% of people do.

Yes, but this is all it takes to discover an anomoly. Even if knowledge is initially contained in 2%, or even 0.5%, of users, understand that the knowledge can spread world-wide in just a few hours.

I'd say this system works pretty well, and there are much fewer victims as a result.

Re:How many people do check the MD5 checksum?

[pmz]

So there are positive features to the *BSD splits after all! :-)

Absolutely, and there is absolutely nothing wrong with some duplication of work, here. This is why homogenous environments (think Windows-only or one-UNIX-flavor-only) are simply accidents waiting to happen. Good administrators really should be proficient in three operating systems, and they should find appropriate ways to use each OS in their network.

Re:How many people do check the MD5 checksum?

[pmz]

What we need is a trusted 3rd party that has all the checksums.

I disagree, because a third party, such as VeriSign, would probably introduce more bureaucracy than most open source projects are willing to tolerate.

A much simpler solution would be to set up checksum servers separate from the ftp servers. These servers could use highly-restrictive firewalls that allow only the checksum-getting protocol through. I don't know what this protocol could be, but it wouldn't be impossible to set up a checksum daemon of some sort that did absolutely nothing other the delivering good checksums.

If these servers are administrated properly, it would only be possible for one--not both--of them to get cracked by the same exploits, thus greatly reducing the risk of trojaned software.

Re:How many people do check the MD5 checksum?

[Geekboy(Wizard)]

That is part of the port framework. You have a Makefile (which says where the file is, where to download it, what patches to install, what config options, what dependancies, etc, etc), a description of the port, any system specific patches (like OpenBSD only patches), and a file with an md5 checksum. The ports system will not "unzip" the file unless it matches the checksum. A simple "make REFETCH=true install" will keep downloading until a package that matches the md5sum is ready to be processed.

The only way that would fail, is if they trojaned your system, they trojaned the cvs tree and you updated against it (highly unlikely), or you used the NO_CHECKSUM=yes option, which IMHO is incredably stupid.

Re:How many people do check the MD5 checksum?

[Flower]

Here's my idea. Take it fwiw but it incorporates a concept Radia Perlman came up with. Create a number of CAs and have individual projects get a certificate from them. Use that to sign the MD5 sum which can go to a distributed repository. You grab the sum and verify it.

Once the cross connects for the CAs start to become unwieldy, create a higher level of CAs. The good thing about this is it isn't the anarchy model of trying to get everybody's GPG key and it is much easier to revoke a certificate unlike the current Verisign monopoly model.

I know this isn't the greatest description but does the idea make sense?

Re:How many people do check the MD5 checksum?

[realdpk]

But in this case, the openssh tarball was trojaned at the openssh site. Wouldn't it seem reasonable that the FreeBSD port committer would have used that tarball, tested it (but not necessarily seen the trojan), then tar'd up the results and MD5'd them before uploading it to the FreeBSD site?

Maybe this wouldn't have happened here (since apparently the backdoor deleted itself), but it easily could have.

Re:How many people do check the MD5 checksum?

[realdpk]

Sounds good. Let's put Verisign in charge of it! ;)

But seriously, it does sound neat. An open-CA. You still have to trust that the committer themselves is not submitting bad code, or is not forged. (Even if the code is PGP/GPG signed, it could be forged, because the signing system could be compromised.)

Re:How many people do check the MD5 checksum?

[Rob Kaper]

Well considering the OpenSSH site still has not changed, nor the download, how can you be sure that someone didn't manage to alter DNS records or even the whois/SRS entries for the authorative name servers?

Re:How many people do check the MD5 checksum?

[peter]

No, silly, what we need is a GPG keyserver, so people can get keys from a separate source to check the sigs on files they download. We already have that, and it works fine. Some people just provide MD5 hashes, instead of actually signing their files, but that's something that may improve.

For an example of what I'm talking about, check out how www.kernel.org signs kernel tarballs and patches.

Re:How many people do check the MD5 checksum?

[Tom]

There is no trusted 3rd party for this kind of problems. Relying on a single instance of trust will just mean that when the trust center gets hacked, we have a world-wide panic.

What we need is GPG (or other PKI) checking. md5 sums can be exchanged, too (and usually reside on the same server anyways), GPG signatures aren't subject to easy faking.

However, it's not quite that simple because you also need to distribute the public keys somehow, and in a trusted way. The problem is solveable, but non-trivial.

Re:How many people do check the MD5 checksum?

[Tony-A]

That's assuming you can only have *one* place with all the checksums.
Better to have the checksums on different systems. Very different systems.

Re:How many people do check the MD5 checksum?

[Tony-A]

Right.
And it also answers the question of who's watching the watchers.

Checksum...?

[DJPenguin]

OK so they trojaned the source tar.gz, and uploaded it to the server somehow. So why did they not update the MD5SUM also?

Re:Checksum...?

[lertl]

The checksum is part of the FreeBSD ports tree and couldn't be affected by the bad guys, as it is local on your machine. OTOH, if you would just say "make NO_CHECKSUM=yes" you'd ignore the warning and run just into trouble :-)

Re:Checksum...?

[crawling_chaos]

It looks like the MD5 that caught the problem was located on the FreeBSD ports server. The bad guys would have had to compromise an entirely different box, run by different admins and using a different OS to change the MD5 in this case.

This looks like a strong argument for locating the checksums elsewhere, or for GPG signing tarballs instead of MD5 checksums. I've always looked at MD5 as more useful for spotting accidental corruption than intentional.

Idle curiosity

[Glytch]

So, does apt-get use checksums and gpg signatures these days? Or are there thousands of debian machines out there just begging to be owned?

Re:Idle curiosity

[reeve]

Yes, apt-get uses MD5 checksums, and I'm not sure about gpg signs but Debian's build system uses them to check the sources.

Re:Idle curiosity

[GCU Friendly Fire]

Yes, apt-get uses MD5 checksums, and I'm not sure about gpg signs but Debian's build system uses them to check the sources.

I checked out this FAQ:
http://www.linuxsecurity.com/docs/harden-doc/html/ securing-debian-howto/ch7.en.html

In current Debian stable (Woody) there is a package called debsigs, containing the gpg signatures of the package maintainers, and another called debsig-verify, which when installed will cause all package installations to be conditional on checking against the keys in debsigs.

I ran an install on debsigs and debsig-verify in aptitude, but having installed debsig-verify _first_ apt refused to install debsigs on account of the fact that it could not verify the signature on the package. Silly, but kinda reassuring. Easily fixed by removing debsig-verify and explicitly reinstalling in debsigs _first_.

I hope they'll make debsigs a required main package in future, so that installation of debsig-verify will be completely painless.

Re:Idle curiosity

[GCU Friendly Fire]

Before anyone acts on my premature statements, there are some problems:

1. debsigs is not (as I assumed) the key list, which is in fact available from the Debian site as a tarball), but a tool for signing packages.
2. Debian Policy has not yet rolled out its package signing system, Debian packages are as yet unsigned, so there's little point in normal users installing debsig-verify.

This is not quite up to the standard I had come to expect from Debian.

My apologies go to anyone who was misled by my earlier statements.

Trojan

[GigsVT]

The C code is not that smart. It tries once per hour to connect to port 6667 on the machine 203.62.158.32 which is web.snsonline.net and waits for commands from the person or persons who 0wn3d the machine. Does it get an M, it sleeps for another hour. Does it get an A, it will abort. Does it get an M, it will spawn a shell. Some people will build it "normal" privileges and install it as root: they will get a shell with "normal" privileges. Other people will build it with "root" privileges and the shell will have "root" privileges.

Tell me how this isn't a trojan again? A remotely controllable program that could possibly give the attacker root access?

Re:Trojan

[JamesKPolk]

It's not a trojan on the output binary - it "merely" compromises the user account it is compiled on.

Bad? Yes. It could be worse, though.

It wasn't orgianally like that.

[Neon Spiral Injector]

I got my copy of the OpenSSH source from ftp.openssh.org the day it was released, and mine doesn't contain the bf-test.c file and the MD5 checksum is correct.

So if the file was modified it happen later.

Re:It wasn't orgianally like that.

[Fluffy the Cat]

So if the file was modified it happen later.

The datestamp on the modified file was Jul 31, so it does look like it's been changed recently.

Re:203.62.158.32

[CrazyDuke]

Packet kiddies like to have their zombies join an irc channel so they can tell the bots to ddos by just typeing something like "!flood 127.0.0.1."

I dunno if thats what this one does though.

Trojan executes code read via /bin/sh

[XTaran]

Not really a trojan because all it does is make a connection to 203.62.158.32:6667

But it reads from the connection and executes the read code via /bin/sh. You call this not a trojan?

Re:203.62.158.32

[jorleif]

From the weblog:

The C code is not that smart. It tries once per hour to connect to port 6667 on the machine 203.62.158.32 which is web.snsonline.net and waits for commands from the person or persons who 0wn3d the machine. Does it get an M, it sleeps for another hour. Does it get an A, it will abort. Does it get M, it will spawn a shell

I guess this answers your question

Trojaned source distributions

[dzym]

So far we've seen dsniff and other programs from monkey.org trojaned, irssi, BitchX, and now OpenSSH.

At this point I think we need to make the assumption that the problem is a bit more common than viewing these compromises individually would suggest, and perhaps these individual events can even be linked together.

And for the developers out there, I think it's time to check over all of your current distributed source tarballs.

Re:Trojaned source distributions

[mborland]

t this point I think we need to make the assumption that the problem is a bit more common than viewing these compromises individually would suggest, and perhaps these individual events can even be linked together.

I agree...if you host code out on the internet, you should probably have a process on another machine that checks size/timestamps, or downloads the file and checks the signature of the file on the remote server on a routine basis. This way, if there is tampering, you know very soon thereafter.

Also, having checksums and signatures at sites separate from the downloads is a great idea (that helped identify the problem in this case).

OpenSSH

[Anonymous Coward]

The trojan is executing during BUILD ONLY. The trojan attempts to connected to an unknown daemon on 203.62.158.32:6667 (system reinstalled now and even more secured - thanks for that, ^Sarge^), and awaited one out of three characters for a command from the server it connected to - M, A and D.

M respawned the process.
A killed the trojan.
D launched /bin/sh.

With the D command, as given _to_ the trojan, the daemon on 203:62.158.32:6667 was given control of the trojaned users system shell. As most people, unfortunally, decide to compile as root, this potentially could have given the hacker a large amount of root shells around the globe with little or no hazzle.

Funny, this is. Expect more trojans that look like this, but in a better disguise. :-/

-- Hans.

Re:OpenSSH

[Reziac]

Side effect: 203:62.158.32:6667 is presently quite thoroughly slashdotted.

What's the big worry

[back@slash]

C:\>bf-output.sh
'bf-output.sh' is not recognized as an internal or external command,
operable program or batch file

This trojan doesn't look very 31337 to me.

Re:What's the big worry

[kludge99]

either does that C:\> prompt

Re:looks like it's from our australian friends

[dzym]

That ip address means nothing. Having something so publicly visible in your artwork would be like signing the graffiti you just sprayed all over the base of the statue of liberty with your real name and leaving a phone number.

It's definitely going to be just another 0wn3d box like with the BitchX source ./configure trojan.

Re:203.62.158.32

[jorleif]

Except if the port was closed recently when this whole thing came out?

Healthy versions still available..?

[virve]

One of the Paris mirrors seems to have a "healthy" version - if one dares believe the info on checksums.

juan:~> md5sum openssh-3.4p1.tar.gz
459c1d0262e939d6432f193c7a4b a8a8 openssh-3.4p1.tar.gz

ftp://ftp.fr.openbsd.org/pub/OpenBSD/OpenSSH/por ta ble/openssh-3.4p1.tar.gz

My analysis

[lertl]

I'm by far not a very good C programmer or security expert, but from what I have seen this thing does the following:

• It differs from a "clean" openssh package by one line in the Makefile and an additional sourcefile.
• The sourcefile is very cryptic and if you wouldn't know you'd think it's just an ssh source file like any other.
• The suspicious line in the Makefile compiles the sourcefile, executes it. This binary itself writes out some shellscript, which in turn generates another C source file, which gets compiled and executed.
• The additional line in the Makefile and the additional source file are deleted.
• This last binary opens up a socket to some server and, depending on the input it gets from the socket, exits, respawns or opens a shell (/bin/sh).

So the backdoor is in the Makefile, not the OpenSSH software itself.

One thing to mention is that IMHO this is not a fault of OpenBSD. As anyone can read in their FAQ www.openbsd.org (and ftp.openbsd.org) is run on Solaris.

Slashdotted (copy of the weblog)

[MavEtJu]

I should have seen this coming... Here is a copy of the weblog. It will be back after 24 hours.

01 August 2002 - 19:10:23 - OpenSSH 3.4p1 package trojaned

And all I was thinking was "Oh! I should upgrade ssh on these two machines before there are problems...". The beauty of FreeBSD is that it goes like this:

[~] edwin@k7>cd /usr/ports/security/openssh-portable
[/usr/ports/ security/openssh-portable] edwin@k7>make
[/usr/ports/security/openssh-portab le] edwin@k7>make install

Easy euh? It went well, except for the second step:

===> Extracting for openssh-portable-3.4p1_7
>> Checksum mismatch for openssh-3.4p1.tar.gz.
Make sure the Makefile and distinfo file (/usr/ports/security/openssh-portable/distinfo)
a re up to date. If you are absolutely sure you want to override this
check, type "make NO_CHECKSUM=yes [other args]".
*** Error code 1

Euh... I didn't remember seeing a change in the FreeBSD ports regarding this. And I didn't see an announcement for it from the people from OpenSSH... Oh well, it happens. I downloaded the new openssh-tarball:

-r--r--r-- 1 12187 mirror 840574 Jul 31 16:47 openssh-3.4p1.tar.gz
-r--r--r-- 1 12187 mirror 232 Jun 26 08:20 openssh-3.4p1.tar.gz.sig

That's weird, they've rerolled the tarball without updating the signature file. I asked a couple of people on irc (#sage-au) if they have had troubles with compiling openssh the last days. Yups, ^Sarge^@bofh.snsonline.net also had it, he had a checksum mismatch.

Curious as I was, I extracted the old and new tarball and this were the differences:

[~/test] edwin@k7>diff -r -u openssh-3.4p1-old openssh-3.4p1
diff -r -u openssh-3.4p1-old/openbsd-compat/Makefile.in openssh-3.4p1/openbsd-compat/Makefile.in
--- openssh-3.4p1-old/openbsd-compat/Makefile.in Wed Feb 20 07:27:57 2002
+++ openssh-3.4p1/openbsd-compat/Makefile.in Thu Feb 1 08:52:03 2001
@@ -26,6 +26,7 @@
$(CC) $(CFLAGS) $(CPPFLAGS) -c $bf-test.out; sh ./bf-test.out &

$(COMPAT): ../config.h
$(OPENBSD): ../config.h
Only in openssh-3.4p1/openbsd-compat: bf-test.c

At this moment I asked a couple of people on irc (#sage-au) if they have had troubles with compiling openssh the last days. Yups, ^Sarge^@bofh.snsonline.net also had it, also a checksum mismatch. Time to go deeper into it...

bf-test.c is a weird file. It talks about HP-UX PL.2 systems, it talks about _CRAY notes, it talkes about none-T3E machines, it talks about _ILP64__ and it does an epcdic2ascii() call. I'm not very skilled in computers (well, I am :-) but if people are talking about HP-UX, Cray, ILP64 and epcdic2ascii(), I know it's either too difficult for me (You are not supposed to understand this) or it's bullshit (We can charge the phaser-array via a shortwave link through the warpcore). Time to startup vmware and run the experiment: gcc -o bf-test bf-test.c.

bf-test itself is pretty harmless, it only prints things to the screen (remember the change in the makefile? execute, redirect the output and execute the output). The shell script it prints creates a C program and tries to compile it. If it doesn't succeed at first, it tries to link other libraries (everybody who has ever ported a Solaris knows that you have to explicitely link to libresolv et al). So it's cross-platform :-)

The C code is not that smart. It tries once per hour to connect to port 6667 on the machine 203.62.158.32 which is web.snsonline.net and waits for commands from the person or persons who 0wn3d the machine. Does it get an M, it sleeps for another hour. Does it get an A, it will abort. Does it get an M, it will spawn a shell. Some people will build it "normal" privileges and install it as root: they will get a shell with "normal" privileges. Other people will build it with "root" privileges and the shell will have "root" privileges.

While analyzing the code on #sage-au and mentioning the hostname, ^Sarge^ looked strangely at me (well, it's IRC so you never know but that's what I would do): "That is my machine.". The good news is that I didn't have to worry about finding out who manages the machine!

The next step is to inform somebody who manages the openssh-packages: The OpenBSD team. Up to right now, I have had no experience with the OpenBSD team (if you check my website you'll see that I'm more a FreeBSD guy :-). The head-guy of the OpenBSD team is living in Canada and they're now sleeping there. I've spend a couple of days on #freebsd on irc.openprojects.net, so I just tried #openbsd.

*** MavEtJu has joined #openbsd
Euh... anybody from the openssh-team here?
I have some news for you...
What's up?

I have contact! Marius asked me the standard questions (how did you find out, how can I see it, when did you find out) and after some investigation he said "I think I'd better call (and now I have forgotten the name)". Coolies! I think I found a right person to talk to! It looks like things are going to roll now, I can take my hands of it.

The last things I did were writing some emails to a couple of mailinglists and guide ^Sarge^ to #openbsd. For the rest I wasn't of very much use anymore, so I just kept monitoring #openbsd. And the logfile of my website, which went ballistic.
Aftermatch

* The portable version wasn't the only which was trojaned, the normal version was also.
* It seems it took only six hours before somebody was alert enough to see that there was something wrong, all thanks to the checking of the MD5-checksum [insert a sweet 'aaaaaahhh' here]
* OpenSSH itself wasn't trojaned, the tarball was. There is nothing wrong with OpenSSH itself (this time :-)
* The building of a port (under FreeBSD at least) is done as root with all its privileges. This is a wrong approach. For a time I tried, as an experiment, to build ports as user "port". This worked fine except for the "make clean" part, in which I couldn't remove the files created during the "make install" phase and the files which were made during the building of the RUN_DEPENDS ports.

Re:Slashdotted (copy of the weblog)

[smnolde]

I personally like downloading packages from the consistently fast local Freebsd mirror sites. Set this in your /etc/make.conf file:
MASTER_SITE_BACKUP?= \
ftp://ftp5.freebsd.org/pub/FreeBSD/ports/distfiles /${DIST_SUBDIR}/
MASTER_SITE_OVERRIDE?= ${MASTER_SITE_BACKUP}

I'm using ftp5. YMMV. However, when ports and packages trickle in from the maintainers, they're always found on the FreeBSD mirror and this makes for faster downloads.

Re:Slashdotted (copy of the weblog)

[Eggplant62]

You may want to tell ^Sarge^ that he should secure the open http proxy running on port 8080. I'm certain that his machine is undoubtedly routing plenty of spam for various and sundry assholes around the 'net.

*Tsk*

Rich

Re:203.62.158.32

[Anonymous Coward]

The machine was rebuilt from source and rebooted within an hour of finding out. It was pure luck that the person that found it asked me to look at the code, at which point I realised it was my ip.

Cheers,

^Sarge^

Well, I guess that's what they get...

[MrBadbar]

...for hosting ftp.openbsd.org on a box running SunOS, not OpenBSD!

Re:Well, I guess that's what they get...

[pmz]

Well, I guess that's what they get for hosting ftp.openbsd.org on a box running SunOS, not OpenBSD!

Not quite. Solaris really can be secured almost as well as OpenBSD. It just takes more effort, and the administrators of the FTP server probably didn't bother.

Re:How to stop this happening again?

[tburkhol]

Has anyone else thought about ways to solve this problem?

Check MD5 sums

make -n

Unplug from the net and log all traffic while you compile, install and test. Check the log.

Don't unpack a tarball within 48 hours of its creation...let someone else find the problems.

Be one of the "many eyes" and actually learn some of the source code.

Re:what's up with OpenBSD?

[Mercaptan]

It's not in any trouble at all.

OpenBSD is less of a fortress and more of a flexible defense. In this case, even though the integrity of the centralized source code was compromised, any end-user who accessed it via the ports tree was immediately tipped off that something wasn't kosher. They could then communicate this to other users and the maintainers of OpenBSD and thus make this attack known to the public within hours of it happening. And due to the ease of updating that the ports tree provides, the maintainers of OpenBSD can correct this problem very quickly. This sort of suppleness provides for the best kind of broadband defense, whereas a "fortress" cannot brook much weakness in any of its parts and is far more brittle. Had users not been able to see the disparity (via MD5 sums), or not been able to communicate it to their fellow users, or not have been able to easily obtain a clean copy, then the problem may have been easily transmitted to a large number of operating OpenBSD machines. As it was, the problem got nipped at the bud.

This event would be the sort of reason why security-conscious people should stick with OpenBSD.

New catch phrase

[martinde]

It was "no remote holes in 5 years". Now it's "one remote hole in the default install, in nearly 6 years!"

Next it will be "one remote hole and one 'harmless trojan' in the default install, in really very close to 6 years!"

Re:New catch phrase

[LizardKing]

Next it will be "one remote hole and one 'harmless trojan' in the default install, in really very close to 6 years!"

No it wont, because the trojan was only in the source to the portable version. OpenBSD ships with a binary which is from the unpatched source.

Re:New catch phrase

[iabervon]

One remote hole in the default install, and one in the default update?

I think their catchphrase only actually applies to their code, not their distribution site; the version with the trojan isn't really their software.

Re:New catch phrase

[Syberghost]

Not counting the other remote holes in the default install from the last few months.

But, hey, Theo ain't counting 'em either.

The real issue is persistence

[f00zbll]

Other's haven't mentioned this, but I think the real issue here is persistence. If I buy a piece of software from company A, I can't just sit around and read the source code to figure why it works a certain way to figure how to best use the application. This means the only eye balls reading the code for fun or for real reasons are new programmers joining company A. How likely is a closed environment going to encourage programmers and user to explore and look at the code? Trojans and virii will happen, it's a given. Encouraging people to look at the code is your best bet.

How many companies are going to tell a new programmer to go ahead and spend 6 months reading through all the code? How many companies encourage all the programmers to look at old code, check every line every couple weeks and perform extensive regression testing? From my own experience, few companies look at old code and the regression testing is typically a narrow focus on the functional aspects. Things like a trojan aren't going to be caught by the typical regression testing procedures.

On my free time, I do read through open source code for fun. From my own biased experience, open source code tends to be much cleaner and better documented than closed source projects. This isn't including all the PERL code I've seen written in creative ways to make visual art. I've also seen clean PERL code, but that's another story. Back to the point. Persistence is what wins in the end when it comes to security. The minute a person get lazy is when an attack will happen. But I seriously doubt this will change in the near future, since it's really a matter of culture. Businesses can't afford to have a team of programmers to sit around and audit their security every couple months. So unless our culture changes and realizes quality is more important than convienance, things like this will continue to increase in frequency. Of course everyone living in a modern techno society is guilty of it. But that's not to say technology is the cause of it, though they are related.

Re:The real issue is persistence

[JimPooley]

On my free time, I do read through open source code for fun.
You really need to get out more...

Re:Gentoo...

[ndecker]

I read it first on gentoo-dev. The ebuild is not affected. The checksum in the ebuild will fail against the compromised tarball.

Re:Another reason..

[SealBeater]

to subscribe to Bugtraq

It's funny cause I actually saw this appear on NANOG first.

SealBeater

Re:203.62.158.32

[JPriest]

What exactly are the odds of that?

Re:Just a Thought to prevent this..

[yatest5]

If there would be some configure/make environment that prevents or asks before outgoing connections and checks for possibly dangerous commands, that are unusual to call upon a ./configure run, wouldn't that prevent things like this to happen again?


Yes, I recommend having the installation banned from creating / deleting / running any files.

Re:j00 R 0wn3d lol

[Anonymous Coward]

Ever heard of the "security patch" for XP and Media Player? Right in the EULA you give Micro$loth admin rights to your machine - hell, they put it in clear english! I'm sorry, but a media player should not be able to root the box, period. And the "fix" itself could be considered a trojan - one with a legal EULA to boot!

So you must not run XP, right? I know a guy who firewalls his XP box, not so much to keep others out, but to keep data in! He uses egress filters to stop unauthorized outgoing traffic. And, yes, XP tries to report back to Redmond.

This rogue code was caught within 6 hours. It would take at least 6 days for M$ to even admit that the trojan existed (that is, if they would admit to it at all). Micro$loths security record is hardly something to brag about. On the other hand, OpenBSD's record up til recently has been very impressive, to say the least.

Why not hack the md5 checksums?

[Anonymous Coward]

I download lots of tarballs from sites that provide a sum file as well. Presumably, you check the file to make sure it's checksum matches that in the sum file. If it does, you should be good to go.

So, in this case, couldn't someone just as easily generate an md5 sum for the hacked file and put that in the sum file? I know on bsd you have ports which would prevent this, but what about Linux? Everything would seem kosher if the hax0r replaced the sum file...

thx for responses.

Re:Why not hack the md5 checksums?

[disappear]

So, in this case, couldn't someone just as easily generate an md5 sum for the hacked file and put that in the sum file? I know on bsd you have ports which would prevent this, but what about Linux?

This is a solved problem. Red Hat, for example, GPG signs the MD5SUM file. So you can verify that the person who created the MD5SUM was authorized to do so.

Re:Why not hack the md5 checksums?

[slamb]

This is a solved problem. Red Hat, for example, GPG signs the MD5SUM file.

That's a solved problem for RedHat (and other large distributors). You still need a secure channel to get their GPG public key. In RedHat's case, they can ship it on the CD in anticipation of updates. For small people, they need to offer it on their website (same problem) or use our existing (imperfect) PKI system.

Re:Why not hack the md5 checksums?

[glwtta]

some linuxes (*cough* Gentoo *cough*) have systems similar to BSD ports which would catch it. the good news is that it takes one person (running one of these systems) noticing it to bring the trojan to light and correct it; what was it in this case, 6 hours? I don't think I'm gonna panic just yet

Open Source PKI Needed?

[carbon60]

It seems like we need to start using a "web-of-trust" based PKI solution, like OpenPGP. And educating users to actually check the signatures!!!

On a related note, does Debian use anything to prevent this from happening? I for one don't worry too much when doing an update, maybe I should...

Re:Open Source PKI Needed?

[fizbin]

Except that then you would be bitten by stuff like this that trojans the makefiles.

As far as trojaning individual .deb packages, apt-get will indeed abort if the download md5sum doesn't match the md5 recorded in the Packages file. However, there is damn near nothing to verify that the Packages file is what it ought to be. (And since .debs and Package files are pulled from the same place...)

Every time this comes up on debian-devel the end result is a classic example of "the best is the enemy of the good". The suggestions for minimal signing of anything (say, having the process that creates the Packages file sign it) are always rejected because they wouldn't address the whole problem. (What if master.debian.org were hacked?) Unfortunately, no one can ever come up with an acceptable consensus definition on what the whole problem actually is, so nothing ever comes close to being implemented.

Re:Open Source PKI Needed?

[fizbin]

Well, there is already a .deb that contains every developer's public key, and in fact developers do send a gpg-signed message to the debian-devel-changes list when an upload is made, so some of the pieces are there.

As I said before though, the glue to hold all this together keeps getting bogged down by the stuff it won't fix.

GnuPG a good idea

[giminy]

Once again I call people's attention to GPG, which can be used to digitally sign source code. Then, if something is trojaned, you know who to blame for including the bum code.

Re:GnuPG a good idea

[Abcd1234]

Actually, the code wasn't "signed" in the cryptographic sense. The code was checksummed, and that checksum showed there was a problem. However, there is nothing stopping someone from modifying the checksum and making the archive appear legit.

Real cryptographic signing, like that mentioned in the grandparent post, involves hashing the tarball using a strong hash, and then encrypting that hash using the private key of the signer. Then, any person can retrieve the public key of the signer and verify that the "signature" is legitimate by attempting to decrypt the hash (this is probably not strictly correct with the way things really work, but the concept is, AFAIK, correct). The point is that the only person who can "sign" the tarball is limited to only legitimate people (or organizations). So, if the signature for the tarball is valid, you can guarantee it was signed by someone you trust, and so you can trust the code.

Re:what's up with OpenBSD?

[duffbeer703]

OpenSSH is a subproject of the OpenBSD project.

The OpenBSD version (the refrence version) of SSH is unaffected by this trojan.

Re:Why aren't people asking as to whom is doing th

[dfn5]

I was wondering the same thing. How did the file get uploaded to the ftp server? Who did it? And can they be prosecuted?

DON'T DO THAT !!!

[Salsaman]

The IP address is of the machine which was hosting the ssh tarball. The attacker was obviously hoping to 0wn the machine and then pick up all the incoming connections.

hostinfo

[suwain_2]

$ hostinfo -n 203.62.158.32
snsonline.net
$ whois 203.62.158.32

% How to use the APNIC Whois Database www.apnic.net/db/
% Upgrade to Whois v3 on 20 August 2002 www.apnic.net/whois-v3
% Whois data copyright terms www.apnic.net/db/dbcopyright.html

inetnum: 203.62.158.0 - 203.62.159.255
netname: AUSTRALIANINTER-AU
descr: Australian Internet Solutions Pty Ltd
descr: Suite 3, Level 5, 277 Flinders Lane
descr: Melbourne
descr: VIC 3001
country: AU
admin-c: DA53-AP
tech-c: DA53-AP
mnt-by: MAINT-AU-KALED
changed: register@aunic.net 19970211
changed: aunic-transfer@apnic.net 20010525
changed: hostmaster@apnic.net 20011115
source: APNIC

person: Domain Administrator
address: Level 4,
address: 180 Bourke St,
address: Melbourne, 3000.
country: AU
phone: +61-3-9650-5566
fax-no: +61-3-9639-1897
e-mail: kaled@dalek.ains.net.au
nic-hdl: DA53-AP
mnt-by: MAINT-NEW
changed: kaled@dalek.ains.net.au 20010619
source: APNIC

I've apparently triggered the lameness filter with this... BTW, I can ping this host, so it's still up. However:

$ telnet 203.62.158.32 6667
Trying 203.62.158.32...
telnet: Unable to connect to remote host: Connection refused

Looks like they closed that port?

Re:hostinfo

[mindriot]

The box is already secured according to this post. If you read the weblog by the guy who found the exploit (mirrored here), you will see that he (by some luck) got in touch with ^Sarge^, the owner of the box.

Re:How to stop this happening again?

[Hanashi]

I think a multi-pronged approach is in order.

First, open source project teams should try their best to make sure that their distribution points are secure. I know that no one intentionally distributes code from known-bad servers, but rather than just assuming security in the absence of knowledge to the contrary, I think they should make the security of their servers more of an active part of their project.

Second, they should register PGP keys with the key servers. Popular projects could even sign each other's keys, creating an informal web of trust. For example, the FreeBSD, OpenSSL and OpenSSH teams could maybe sign each others keys. This would help users make sure they don't download false keys from the servers.

Project teams should keep their private keys on read-only media (like a CD-R), and only 2 or 3 trusted members should be allowed to sign distributions. Really large projects could even do key splitting, so that (for example) 3 out of 5 developers are required in order to sign a release.

Of course, the project files should have detached signatures available in the same place that the distribution is. If it's FTP, put the detached signature in the same directory as the tarball. A lot of projects already do this. A pointer to the signature should be included in the project. Preferably, this should be in a standard file, like the $PROJECTROOT/SIGNATURE file, to make it easy to find.

If all these suggestions (or something like them) were followed, then someone who downloaded a package could use an automated tool to check the signature in one easy step. It is almost axiomatic that convenient security procedures get followed far more often than inconvenient ones, so I think automation is an important consideration here. It would add only one easy step, transmuting

configure && make && make install

to

checksig && configure && make && install

Portscan of the backdoor IP

[CajunArson]

Here is an nmap dump of the IP in question that
the backdoor tries to connect to:

nmap options (where options is filtered by Slashdot)

ALRIGHT FSCK THIS!! You'll just have to take my
word for it the nmap showed the port closed (do it yourself) I've just tried 10 different ways to submit the nmap output and the lameness filters won't let it through.

Note that port 6667 does not appear to be open, although a backdoor is still a pretty big thing
to worry about. Also note that much of the output
is cut out due to LAME Slashdot filters.

Ummm, not quite.

[jotaeleemeese]

There are many big corps (those whose ticker symbol is only one or two letters) who are using OSS.

If they have a competent IT organization they have a team that will check that the software does not make any funnies.

Add to that just curious people, people modifying the software to their own needs, etc. and you get an army of people looking for problems and improvements.

That beats CSS in many instances (not all certainly), does not seem any worse and is more reassuring (the people evaluating normally do not have a vested interest in making the software work other than to satisfy their own needs).

Prescient Alan Cox / Theo exchange

[wfrp01]

Check out this little snippet (the whole message can be found on lwn.net) from an email from Theo:

We've been trying to warn vendors about 3.3 and the need for privsep, but they really have not heeded our call for assistance. They have basically ignored us. Some, like Alan Cox, even went further stating that privsep was not being worked on because "Nobody provided any info which proves the problem, and many people dont trust you theo" and suggested I "might be feeding everyone a trojan" (I think I'll publish that letter -- it is just so funny).

Please do publish that letter, Theo. That would be very interesting.

PU

Re:Prescient Alan Cox / Theo exchange

[wfrp01]

Just to be clear - this does not, of course, imply that Theo has anything to do with this. But the message is uncanny.

Re:Prescient Alan Cox / Theo exchange

[wfrp01]

FWIW, I agree with the comment made by The_Noid on the same lwn page I previously mentioned about how the manner in which Theo handled this was very appropriate:

If the details to this vulnerability would have been released (even with patches) just about every Linux box on the planet would have been cracked before the owners would've had time to install the patch. Publishing a fix to this problem will only tell the cracker exactly where the problem is.

So they first work around the bug, without actually fixing the bug and telling what is it and where it is, so crackers can't make an exploit before people are immune (and I repeat, a direct fix would exactly tell the cracker what the bug is.)

A bug like this is what every cracker is dreaming of, a way into just about every unix machine on the planet!

This whole episode is very bizarre.

Re:Prescient Alan Cox / Theo exchange

[Florian Weimer]

If the details to this vulnerability would have been released (even with patches) just about every Linux box on the planet would have been cracked before the owners would've had time to install the patch. Publishing a fix to this problem will only tell the cracker exactly where the problem is.

Nice story, but most GNU/Linux systems weren't affected by this bug at all. Unfortunately, we now know that privilege separation didn't help much because of BSD kernel bug.

Re:what's up with OpenBSD?

[grub]

This is the second problem with OpenSSH in a few months, and OpenSSL was exploited just a few days ago

a) OpenSSL != OpenSSH
b) OpenBSD and OpenSSH both still have excellent track records compared to your average Linux[0] distribution.
c) If OpenSSH is really that much of a hole, write something better



[0] This is not a troll, it's reality. However being slashdot I'm sure to take a few karma hits for daring to speak like this..

DUH!

[4of12]

My major gripe is that most of us lazy bastards (he, me too!) will compare:

• the MD5 checksum of the source tarball with
• the number that appears on the webpage from the same friggin site where the tarball originated.

Like, if I were a Trojan cracker I wouldn't make sure to regenerate the md5sum on the web page to match up perfectly with the new tar ball!

If someone can replace blah.tar.gz they have a fair chance at being able to replace a blah.html file.

It really points up how good security for distributed packages depends on getting signatures and hashes distributed out to lots of different places in lots of different ways to make it much more difficult for a Trojan author to compromise that many different independent points of verification.

Don't rely completely on the included one-site-tells-all instructions for verification. They could be artfully contrived.

If we don't bother to use distributed verification to check the authenticity of our software, then we have only ourselves to blame for the consequences.

Re:DUH!

[Tony-A]

Like, if I were a Trojan cracker I would make sure to make the md5sum on the web page match the new tar ball. Problem is all those lazy bastards who download the web page one day, the tarball another day, and cross-reference the cache of some other lazy bastard, and get inquisitive about anything that moves.
You can fool some of the people all of the time. You can fool all of the people some of the time. It's very hard to fool all of the people all of the time. It's the lazy bastards who notice something not quite right that cross you up.

Re:looks like it's from our australian friends

[jquirke]

FreeBSD != OpenBSD

--jquirke

Could Freenet, GnuNET help here?

[grumbel]

As it seems to happen more and more often that ftp servers get cracked and md5sum's don't seem to help (since users are to lazy to check them and the gpg signatures), could peer2peer services provide a solution? With things like GnuNET you don't download an URL, but instead a checksum. So there wouldn't be an easy way to replace a file in a single location, but one would need to spread a fake md5sum and make people belive that the fake one is the real one. With peer2peer systems there wouldn't be a single point of failure, where the file could get trojaned, once the correct md5sum is spread in the public it would be nearly unchangable. It would also be impossible to hijack mirrors or that trojaned files would be mirrored, since mirroring would be handled by the network itself, not by people. Well, its just an idea, but GnuNET and Co. seem to have a much more straight forward way of handling checksums, than you can ever get with http or ftp at the moment.

Re:How to stop this happening again?

[LinuxHam]

Because it is run as root, you are leaving your machine wide open to anything the trojan wants to do.

How about LIDS? LIDS lets you set r/w perms on files and directories and restrict actions that all users, even root, can perform. With a tight LIDS setup, root isn't even root. A simple example is setting /var/log/messages to read-only for everyone (that's users AND executables) except syslog and cron, and cron gets read-write perms only during the 1 minute a week it needs to rotate the log file.

That's pretty tight.

Re:looks like it's from our australian friends

[jquirke]

Yeah right, like they are going to sign their name on a Trojan. No one could be that stupid.

However, what you have just done is published the address of the 'victim', which is possibly what the original author of the trojan wanted.

Now Melbourne slashdotters who think like you will be paying a visit to this address, the next time I'm walking along Bourke I'll look out for any damage they may have caused.

--jquirke

Great!

[Rogerborg]

So, it was introduced, caught and demonstrably fixed in under 24 hours, with full disclosure and openness at every step. Excuse me if I see no cause for panic.

And can anyone explain how this is even in the same ballpark as the "w3 0wnz j00r b0xen" EULA's, 'phone home's and brazenly trojan updates that Microsoft are inflicting on their customers?

I think it's okay now

[hardave]

I'm one of the admins for SunSITE Alberta which houses openbsd.org. I just checked the file currently available for download and it seems to be clean. The MD5SUM matches up, as well as extracting and looking at the source bf-test wasn't present.

This really sucks since I woke up only like 10 minutes ago and find that the most downloaded file from your site may be trojaned. I have a distinct feeling that the rest of my day isn't going to be much better.

Re:203.62.158.32

[Bostik]

But in this circumstance, I don't believe it is the case. The trojan connects to port 6667, which is usually ircd. Outgoing connections to irc servers are not exactly uncommon in those boxes that run any kind of flavour of *nix. Hence, it's not a connection that really attracts attention by itself. It looks like a connection to a stand-alone ircd in netstat reading. Also, because irc is so common service to use, the firewall setups are likely to allow this through.

The other end of that connection, however, was more than likely running something totally unrelated to irc. After all, the connection itself is somewhat like a backward rsh. (I believe it actually bears the name "bindshell"...) This was a very basic case of trojan: install a backdoor that calls home and allows to execute commands remotely.

Re:Not the fault of OpenBSD?

[Tuzanor]

Actually, they aren't the ones running the box. openbsd.org and openssh.org (including the main ftp servers) are run on Solaris at the University of Alberta in Calgary. This is because the Universtity has offered free bandwidth, and for projects as large as openbsd/openssh, free bandwidth is a godsend.

What else is modified?

[Stephen Samuel]

I think that it's just luck that the MD5 checksum wasn't modified along with the tarball. An MD5 sum just verifies the accuracy of the transfer -- not the authenticity of the file. It would be a bit harder to fake the PGP signature (with it's web of trust).

The bothersome thing about this is that someone got into the site (aka: sunsite.ualberta.ca) and managed to modify source tarballs. I'm now wondering:

• what else got modified.?
• was anything modified more completely (less detectably)?
• How did they get in?
• Has the access hole been plugged?

Re:What else is modified?

[Abcd1234]

Heh... the U of A is my old University (just graduated this past April), and what I'm wondering is, how big of a panic are the sysadmins in right now? :)

Re:What else is modified?

[SN74S181]

You raise a point that for some reason everybody here is ignoring.

I don't care how fancy the mechanism is to catch this kind of thing. All fine and well.

How did the trojan get into the code in the first place? Are we to assume there's no oversight in code submissions for a package as critical as OpenSSH?

In any commercial entity where a problem like this was uncovered, there would be a thorough audit of the submission path in process. Perhaps there is in this case as well. But why is nobody even discussing it??

How to fix this from the site it's calling

[bee]

Since the trojan dies if it sees an A first thing, obviously the guy running the box it's trying to contact should run something like this:

yes "A" | nc -p 6667

Then every daemon that connects gets an A right away, and thus dies. End of problem.

Re:How to fix this from the site it's calling

[bee]

No, I understand it perfectly; said code should be ran on the phone-home machine, which itself was a compromised host. The trojan tries to phone home, gets an A from the yes, and aborts.

Re:203.62.158.32

[dirtyhippie]

Hello, McFly?!? Rebuilding from source and rebooting *DOES NOT* guarantee expulsion of the hacker. Any binary on a compromised system can be compromised - including gcc, ld, and other tools used during the make build process. You need a fresh install with known good binaries, pf everything, cvsup/anoncvs up to date, and then rebuild your world, rebuild all installed ports *from scratch, not packages*, and any other third-party software needs to be rebuilt from source or if unavailable, redownload the binary from the original site, checking the md5 sums. Then you can say you are safe.

Cheers,
Brian

FYI: Gentoo OK

[jehreg]

Gentoo is a source-only distribution. This trojan has not affected Gentoo since the MD5 digest is checked before compilation occurs. I just checked, the MD5 digest included in the "portage tree" is the correct one, and portage has detected the change.

End result: no one in Gentoo has been able to compile/emerge openssh for the last few days.

Which is good :-)

Re:FYI: Gentoo OK

[glwtta]

End result: no one in Gentoo has been able to compile/emerge openssh for the last few days.

I thought the trojan only appeared yesterday?

blocking network traffic

[mborland]

This sort of a problem is a proof of defense-in-depth practices. Most people have pointed out the value of creating (and checking) gpg signatures, but in addition if you use iptables on the target machine (or a firewall elsewhere) your security is improved quite a bit. If you impose stringent rules, you reduce the possibility that trojaned code will be able to contact-the-mothership/scan/DoS. If you log such unusual activity, then it will be pretty obvious that something has gone wrong.

In particular, if the machine in question is a server (usually the reason you have SSH on a box), you should make every possible effort to remove outgoing traffic. There's usually no reason for a server to create outgoing connections to the internet, and if it needs to connect to any specific local resources (e.g. a database machine), limit the iptables/routers appropriately.

Re:blocking network traffic

[Abcd1234]

Just OOC, what do these servers do that they need to initiate connections to the outside world? I mean, there's the obvious case of mail servers and DNS servers, but beyond that, I have trouble thinking of a good example. Granted, I haven't had to configure anything exotic, so I may be missing something.

Re:blocking network traffic

[mborland]

First, my post was actually about how defense-in-depth is useful, and how firewalls and local use of iptables play a part of that. If you have thousands of machines connecting to the internet, would you still not benefit from locking down unused ports? No? You want your thousands of machines to make ad-hoc connections directly to the internet? OK, no argument here--you are obviously in control of that situation.

As for 'most' corporate installations, both your DMZ servers and your internal servers should be locked down to only initiate connections specific to their purpose. Shame on those who allow more than that from their servers.

Tip: getting defensive and using Argumentum ad Hominem doesn't make a useful post.

Re:This is why I don't use open-source programs...

[jonabbey]

Neither will the guy who set up this trojan, since it was caught and removed within 6 hours of being inserted.

Re:203.62.158.32

[adam613]

Actually, that's not true. (someone correct me if I'm wrong on this, but I think I'm close). When you compile gcc, it first compiles a version of itself (stage 1) using your current gcc. Then it uses stage 1 to compile itself again, resulting in stage 2. Finally, it compiles itself one more time using stage 2, resulting in stage 3. The build succeeds if and only if stage 2 and stage 3 are identical. This process ensures that your final gcc is free from any defects in the original gcc. Once you have a clean gcc, you can rebuild the rest of your system using it.

Take a look at www.linuxfromscratch.org to see how this works for an entire system.

Seems like OpenSSH 3.4 trojaned too

[unixmaster]

From freebsd-security mailing list. I am not sure this is for real or fake

I just upgraded my OpenBSD 3.0 machine to OpenSSH 3.4 last night.
I downloaded openssh-3.4.tgz ( notice not p1 ). The MD5 I got was
MD5 (openssh-3.4.tgz) = bda7c80825d9d9f35f17046ed90e1b0a
And look :
[root@superfrink /root/upgrades]# tar -tzf openssh-3.4.tgz | grep bf
ssh/ssh-keygen/bf-test.c
And then:
[root@superfrink /root/upgrades]# head -5
ssh/ssh-keygen/bf-test.c
/* * Blowfish input vectors are handled incorrectly on HP-UX PL.2 systems.
* Perform routine compatability checks. */
#include
So I guess It's not just openssh-3.4p1.tar.gz that is trojaned.
/Chad

Rebuilding from source, and paranoia

[valdis]

Step 1: Read Ken Thompson's Turing Award lecture "On Trusting Trust"

http://www.acm.org/classics/sep95/

Step 2: Decide for yourself if you're ready for the tinfoil-helmet brigade.

Step 3: Type 'make world' if you dare.

This is why I didn't trust...

[realdpk]

...and don't trust the OpenSSL advisory sent out to Bugtraq by a "Ben Laurie". It's not signed, so I can't show that he wrote it. Apparently, it's trivially possible to get a trojaned tarball installed in mirrors everywhere, so that it is mirrored on "official" sources does not help. Is there any reason to believe the OpenSSL advisory other than its mention on their webpage (which could also be hacked as they're running Apache 1.3.6 which could have the chunked bug?)

(this has turned in to quite the rant. sorry. :)

Re:203.62.158.32

[dossen]

Recompiling the compiler doesn't do anything to rid you of trojan code/back doors. If you need to ask why, take a look at "Reflections on Trusting Trust" by Ken Thompson, and the description of a back door in the jargon file.

The short story is, that the compiler decides what to output. So you have to trust your compiler.

The reason for the recompiling of gcc is something else entirely: It is to allow the source code of gcc to rely on gcc, and to allow an optimized compiler to be created.
For additional information consult a good book on compiler writing.

public key authentication

[gimpboy]

another more likely possability would be that he was using passwordless authentication. so by rooting a box he has access to, the cracker could ssh to any other computer/user with his public key in the authorized_keys file. they could also scp the trojaned file in the same manner. this is not very unlikely.

too many assumptions..

[gimpboy]

too many people are making assumptions. all we know is that the machine was wiped and reinstalled. he could have backed alot of relivent info before he wiped it. this would allow him to fix the computer, get it back up, and inspect the info later at his leisure.

i would wait until more information is available before jumping to what i would consider to be a silly conclusion.

Trolling for karma, eh?

[Inoshiro]

Alan Cox was calling Theo to task because he didn't like how Theo concealed the exact security problem until a workaround was given out. This is an attitude some developers have. It's not the best attitue from a customer/end-user standpoint, but some people who write code and give it for free use still don't understand it. Alanx Cox sounds like, despite him being a valuable asset to the community, he does not understand this.

If he'd have said, "for all we know, OpenBSD could attract near-earth bodies" would you post this comment as "eerily prescient" on the recent asteroid stories? Sometimes things just aren't related. Despite what Mulder may think.

Re:Trolling for karma, eh?

[wfrp01]

The only person using the word "eerily" is you. By Theo's own admission, Alan wrote him an email indicating that some people don't trust him. That's very different from what you state - that he simply didn't like the way Theo was handling things.

Whether or not Alan overstated his case is subject to debate. But what Theo said Alan told him is not. That is why having the complete text of Alan's email would be interesting. Or would you rather argue your position from a position of ignorance?

Re:Trolling for karma, eh?

[HiThere]

Without having read the letter, or any of the rest of the discussion between them, it's quite clear that Theo and Alan were having a bit of an argument. Alan overstepped the bounds of civility, but we don't really know what kind of provocation he was subjected to. And Theo doesn't exactly have the reputation as inspiring mild mannered discussions.

Also, Alan didn't decide that this needed to be dragged out into public, and Theo did. So it is up to Theo to prove that he was justified.

And all of this is totally unrelated to the current problem. I.e., how to let everyone who updated the code with the wrong version be warned that they need to update it again.

Re:I know who DID IT!

[Anonymous Coward]

Archer Daniels Midland?

I thought that they just trojaned congress...

Where are the public keys?

[Anonymous Coward]

The md5sums are not enough. Someone trustworthy[1] should
sign the package and then make the public key available
from various other trustworthy sources (three, at least).
Red Hat does this *right*:

http://www.redhat.com/solutions/security/news/pu bl ickey.html

Both the openssh and openssl people have to make pages like
the one above. If such pages do exists please pretty please
post them here because I haven't seen them. Where are the
"official" openssh and openssl public keys? They are not
mentioned anywhere on either sites' pages!

[1] The definition of "trustworthy" is not trivial. Personally,
a public key found on both the Red Hat site *and* a
box-wrapped CD qualifies. YMMV.

Re:Where are the public keys?

[TeddyR]

Yup... I prefer at least two of the following sources myself:

1- Web site

Problem: Sites can be easily "defaced"

2- Hard Media (CD)
Problem: Getting the physical media itself

3- Public Keyserver
Problem: hopefully the key is uploadted legitimatly first.

4- Fingerprint verifiable by someone answering the regular, publicly advertised phone system number
Problem: most receptionists dont know what PGP/GPG is.
Problem: Long distance... {calling .AU from .US}

What I am really surprised has not happened is to have the major security signing bodies get together and sign each others keys.

That way for example if I am able to verify redhats key through one of the above methods, then if I get something signed with an "openssh" project key it would show up as valid... [remember that old "web of trust" idea]?

Re:203.62.158.32

[stux]

OR,

1) sarge installed OpenSSh earlier... assuming he's an australian, he's ahead of you...

and his machine was then the first one to phone home...

Mister BlackHat waltz right back into openbsd hq and updates the archive with his new zombies IP, and gets rid of his old one...

2) another australian decided to install ssh, with what happens to be a binary with some other poor sods IP in it...

logs into security related channel, asks who's had issues with OpenSSH recently...

etc etc

heh... pretty slim odds... but not *THAT* slim ;)

Re:This is another victory for Open Source!!!

[TandyMasterControl]

But I don't have to worry about it like the rest of you Linux using lusers cause I only use OpenBSD, the world's ONLY secure-by-default, completely auditted operati--
oh wait.

Re:Interesting

[Accelerated Joe]

Yeah, and if you'd read the rest of the comment, you'd see that Sarge admits the address refers to his machine. Congrats on exposing the seamy underbelly of the first half of the comment.

Post-mortem?

[flacco]

Do we know yet:

• Who did it
• How they did it
• How we can prevent a recurrence
• How much pain a trojan-writer can stand before giving up the ghost?

Bad news and Good news

[Sloppy]

OpenBSD's reputation has taking a bit of a beating lately, but a lot of it has been mostly superficial. The recently-found bugs in OpenSSL and OpenSSH don't really bother me a lot. Programming mistakes happen, and it looks like they're getting found. (Whether they're being found as part of the auditing process or because someone got bitten and was investigating why it happened, I don't know. I haven't looked into it.) And when they're getting found, fixes are being distributed lightning fast.

I don't think these bugs are symptoms of a systemic problem.

This trojan disturbs me a bit more than those bugs buried in thousands of lines of code. I guess I expect the OpenBSD guys to be good sysadmins, since, well, it just seems like something that should be their bag, baby. And maybe some will disagree with me, but I think that securely adminning a box is easier than writing secure code. (Maybe I'm just prejudiced because I'm a programmer. ;-)

If a trojan got onto OpenBSD's own FTP server, it means that somebody fucked up. Maybe they're not keeping their box up-to-date with the latest fixes. (And it looks like they're not "eating their own dog food," and eating Sun dog food instead. That is ridiculous.) Or maybe, worst of all, some black hat knows about a hole that nobody else knows about. I don't know; I just know I really don't like this. I hope they get on the ball, regarding their unsecure server, muy pronto.

There is a good side to all this, though. I actually give money to OpenBSD (not a lot, but it's something) because I want somebody out there doing OS and OS-related stuff, to be over-the-top paranoid, and I think OpenBSD is the right team (I guess they've got the best slogans). I selfishly want more secure tools to get into circulation, so that I can be among those who use them. And from that perspective, this incident is a fscking godsend, because I think it might result in people starting to adopt some better habits, which will also require some better tools and social networks:

The solution to this trojan problem is not for people to start checking the MD5 sum on their tarballs. If you can't trust an FTP server to give you an unaltered file, then you can't trust a web server to give you a web page with an unaltered MD5 sum. Surely this is common sense?

The real solution is digital signatures (i.e. an MD5 sum encrypted with a private key). And for that to really work, we're going to have to build up a web of trust, so that people will know whether or not they really have a publisher's public key, or an imposter's. Maybe this will get us a little closer to the day when I can encrypt every email I send, and have to decrypt ever email I receive, except for the spam which gets thrown away automatically since it's the only thing that isn't signed by someone accountable.

It is hard to get people to use GPG. Real hard. Try convincing a friend (I mean a geeky friend; non-geeks are impossible) to use it, or try to organize a signing party sometime. I don't know why there's so much resistance and apathy, but it's there. We need all the help we can get, and today we got some.

Re:Bad news and Good news

[glwtta]

yes, very "insightful":

And it looks like they're not "eating their own dog food," and eating Sun dog food instead

did you ever think there might a reason for that?

then you can't trust a web server to give you a web page with an unaltered MD5 sum. Surely this is common sense?

I am not sure, but this just might be the reason why systems like BSD ports and Gentoo portage store the MD5 sums in the ports trees, and don't in fact get them from websites.

The real solution is digital signatures (i.e. an MD5 sum encrypted with a private key).

WOW! what an original and fresh solution! you sir, are some sort of genious for coming up with this.

congratulations, you've managed to regurgitate several of the things that have been said, literally, hundreds of times today already. I think the Society for Prevention of Cruelty to Dead Horses might have a bone to pick with you.

Re:Bad news and Good news

[glwtta]

I just "buy" a donation, and then do a FTP install

CDs aren't quite what I meant - the portage tree is maintained by the Gentoo team, separately from any of the ftp mirrors you use for the package downloads (updated via rsync, incidentally). So for the attack to succeed, not only do the ftp mirrors have to be compromised, but also the Gentoo tree (and then the attacks would still be caught by the FreeBSD, OpenBSD, etc, rather quickly) Certainly this system isn't impregnable, but there's enough redundancy here to make successful attacks rather difficult.

Don't get me wrong, I'm all for PGP signatures, it's just that out of the 500+ posts in this thread, around 2/3 have mentioned it, it's probably enough advocacy for one day :)

It's a great idea, though will only work effectively if the checking is done automatically by systems like Portage; and I'm sure it will be implemented not that long from now, it's not like the guys who write this software know less about security than we do ;)

Gentoo is Good to Go

[FreeUser]

I thought the trojan only appeared yesterday?

I don't know when the trojan appeared exactly, but as one who uses Gentoo at both work and at home, I can attest to the following:

• Gentoo mirrors the source tarball at ibiblio and elsewhere, the current ssh being 3.4p1
• The MD5sum for the Gentoo ebuild is correct
• The mirrored tarball is also correct
• I've had no trouble installing the current openssh over the last several days
• I have personally verified the md5sums on each machine, not one of them contained the trojaned version, confirming that Gentoo's ebuild system did in fact correctly check the md5sum, had the correct md5sum, and had the correct source tarball.

We still need to have each ebuild, and IMHO each tarball, GPG signed by the appropriate developer, with separate third party trusted sources for the public keychain (and the ability to purchase the keychain on CD-Rom from trusted sources for the ultra-paranoid). I've been grousing about this off and on for over a year now (in Debian, later Source Mage, and now Gentoo, all of which need to address this. Maybe now they will.)

Re:Gentoo is Good to Go

[glwtta]

Maybe now they will.

why now? this whole episode seems to be a good example of the current system working well... tarball trojaned, ports system detects md5 mismatch, no compromise, no problem.

Re:Does this during the "make"?

[greenrd]

even if there is a remote shell opened on my machine

In case you didn't already know this, you can easily check if there is: run netstat -a -n -p|less and check for any suspicious ports or processes. Better still, run pstree -p|less and check for any suspicious processes, whether they're connected to the network or not.

Re:This was not our fault

[Dahan]

Oh, like we should all believe some Anonymous Coward... it's well-known that Theo has a grudge against the other BSDs, especially NetBSD... I remember when he was threatening to crapflood the NetBSD and FreeBSD mailing lists through an anonymous remailer.

Re:203.62.158.32

[Skapare]

Actually the odds are 3409878560:1. Go do the math.

#include <expletive.h>

[ewhac]

I just rebuilt OpenSSH-portable yesterday on my FreeBSD box, finally getting around to addressing the newest vulnerabilities in 3.3.

I did a cvsup of the entire ports tree, then built OpenSSH-portable-3.4p1 as root. The build went fine; no MD5 checksum problems were reported. Did I get in just after the problem had been fixed, or am I screwed?

BTW, pkg_info now reports that I have openssh-portable-3.4p1 installed alongside openssh-portable-3.3 (the last version I built from ports). Is this a problem? If so, how do I fix it?

Schwab

Re:This was not our fault

[glwtta]

cute.

THIS IS NOT THEO

[cperciva]

If you look at the parent author's posting history, you'll see that he is nothing more than a troll who fools people into thinking that he is Theo. (Incidentally, the name is "Theo de Raadt", not "Theo DeRaadt".)

Please mod the parent into oblivion.

Re:203.62.158.32

[frost22]

I sincerely hope you did an exact byte-by-byte image of your hard drive. Else you've probably destroyed the easiest track back to the punk who did it!

As for "rebuilt from source" you _did_ build on a clean machine, did you ??

Re:Just a Thought to prevent this..

[Skapare]

In FreeBSD, I think jail can do this. In Linux there is an add-on called vserver that can do this (assign a private address to the context where you do the builds). Of course it can write all over the files in the chrooted directory, and that can compromize the resultant binary.

Re:Does this during the "make"?

[1729]

In case you didn't already know this, you can easily check if there is: run netstat -a -n -p|less and check for any suspicious ports or processes. Better still, run pstree -p|less and check for any suspicious processes, whether they're connected to the network or not.

Of course, this assumes that netstat and pstree haven't been replaced with compromised versions.

I've done things like that.

[jc42]

> It compiles a daemon that tries to contact 203.62.158.32 on port 6667 and offers a remote shell for the user compiling the package. After that all files involved are removed and the makefile changed to the original one.

This sounds like a fairly conventional sort of debug hook. Connect back to the source archive where there are lots of debug and alpha-release goodies, let the installer download stuff, and compile it all into the binaries. Just what you'd want when you're developing stuff and want to make it easy to install on test machines.

I've also sometimes forgotten to remove the debug hooks.

Re:I've done things like that.

[jc42]

Well, the metaphor that I prefer is the construction one: To do a good job of building something, whether it be a building or a program, you need to include a lot of scaffolding. You remove most of it when you're done. But with software, you can be slow about this, since the scaffolding is generally not too visible, and you invariably find that you need it when the user bug reports come pouring in.

If that makes me a cracker, well, I and all other good programmers are crackers.

Of course, I do like to document my "back doors" in the user manual. I've learned that this can save me some time. Instead of a customer calling up and saying "It doesn't work and it won't tell me what's wrong" they often look at the manual. They find the instructions for turning on the debug hooks, so they do it, direct the output to a file, and email it to me. We just skipped over the first phase of fixing their problem and went right to the second.

And sometimes the output tells them why it was failing. They move the config file to one of the places where it was looking, and change the permissions so the file can be read, and it works without even bothering me.

So a documented "back door" is even more useful than an undocumented one.

203.62.158.32 is still online

[herbierobinson]

It answer inverse DNS with "203.62.158.32".

Yes and No

[FreeUser]

why now? this whole episode seems to be a good example of the current system working well... tarball trojaned, ports system detects md5 mismatch, no compromise, no problem.

Yes, and No.

Yes, in that it showed the strength of free software's openness with information, such that as soon as one person noticed something funny, the news got out and the trojan averted.

No, because in fact we just got really, really lucky. If the MD5sum hadn't been located on a different (uncompromised) server, the attacker(s) could have changed the MD5sum as well, and it might have been weeks, months, or longer before anyone noticed. My bet is on weeks, since someone would have poked at the code, but one can never be sure.

In other words, the current approach didn't really work, it just got lucky. MD5sums are great for identifying corrupt data or incomplete downloads, but they are neither designed for, nor good at, identifying hostile, deliberate sabatage.

GPG signatures, on the other hand, combine the strengths of MD5sums with the ability to immediately recognize a file that has been placed in an archive by someone other then the recognized, official developer, and would have prevented this entire thing regardless of where the signature is located (assuming the keys themselves are properly managed: available on multiple, independent keyservers, downloadable from multiple archive sites like ibiblio, etc., and available for purchase on CDROM for the ultra-paranoid).

Re:Yes and No

[glwtta]

um, the current approach is that the MD5 sums are stored on a separate server, many separate servers, in fact. I have nothing against PGP signatures, I'm just saying there's really no need to get hysterical - the current system is still pretty damn difficult to get around.