Slashdot Mirror
CERT: Sendmail Distribution Contained Trojan Horse
Scoria writes "According to a CERT advisory published this afternoon, the public distribution of Sendmail 8.12.6 contained a trojan horse from September 28 to October 6. For more detailed information, please consult advisory CA-2002-28." This sounds very much like what happened to OpenSSH.
CERT® Advisory CA-2002-28 Trojan Horse Sendmail Distribution
d 9d2137 sendmail.8.12.6.tar.Za fef34 sendmail.8.12.6.tar.sig
t ml
Original release date: October 08, 2002
Last revised: --
Source: CERT/CC
A complete revision history is at the end of this file.
Overview
The CERT/CC has received confirmation that some copies of the source code for the Sendmail package were modified by an intruder to contain a Trojan horse.
Sites that employ, redistribute, or mirror the Sendmail package should immediately verify the integrity of their distribution.
I. Description
The CERT/CC has received confirmation that some copies of the source code for the Sendmail package have been modified by an intruder to contain a Trojan horse.
The following files were modified to include the malicious code:
sendmail.8.12.6.tar.Z
sendmail.8.12.6.tar.gz
These files began to appear in downloads from the FTP server ftp.sendmail.org on or around September 28, 2002. The Sendmail development team disabled the compromised FTP server on October 6, 2002 at approximately 22:15 PDT. It does not appear that copies downloaded via HTTP contained the Trojan horse; however, the CERT/CC encourages users who may have downloaded the source code via HTTP during this time period to take the steps outlined in the Solution section as a precautionary measure.
The Trojan horse versions of Sendmail contain malicious code that is run during the process of building the software. This code forks a process that connects to a fixed remote server on 6667/tcp. This forked process allows the intruder to open a shell running in the context of the user who built the Sendmail software. There is no evidence that the process is persistent after a reboot of the compromised system. However, a subsequent build of the Trojan horse Sendmail package will re-establish the backdoor process.
II. Impact
An intruder operating from the remote address specified in the malicious code can gain unauthorized remote access to any host that compiled a version of Sendmail from this Trojan horse version of the source code. The level of access would be that of the user who compiled the source code.
It is important to understand that the compromise is to the system that is used to build the Sendmail software and not to the systems that run the Sendmail daemon. Because the compromised system creates a tunnel to the intruder-controlled system, the intruder may have a path through network access controls.
III. Solution
Obtain an authentic version of Sendmail
The primary distribution site for Sendmail is
http://www.sendmail.org/
Sites that mirror the Sendmail source code are encouraged to verify the integrity of their sources.
Verify software authenticity
We strongly encourage sites that recently downloaded a copy of the Sendmail distribution to verify the authenticity of their distribution, regardless of where it was obtained. Furthermore, we encourage users to inspect any and all software that may have been downloaded from the compromised site. Note that it is not sufficient to rely on the timestamps or sizes of the file when trying to determine whether or not you have a copy of the Trojan horse version.
Verify PGP signatures
The Sendmail source distribution is cryptographically signed with the following PGP key:
pub 1024R/678C0A03 2001-12-18 Sendmail Signing Key/2002
Key fingerprint = 7B 02 F4 AA FC C0 22 DA 47 3E 2A 9A 9B 35 22 45
The Trojan horse copy did not include an updated PGP signature, so attempts to verify its integrity would have failed. The sendmail.org staff has verified that the Trojan horse copies did indeed fail PGP signature checks.
Verify MD5 checksums
In the absence of PGP, you can use the following MD5 checksums to verify the integrity of your Sendmail source code distribution:
Correct versions:
73e18ea78b2386b774963c8472cbd309 sendmail.8.12.6.tar.gz
cebe3fa43731b315908f44889
8b9c78122044f4e4744fc447ee
As a matter of good security practice, the CERT/CC encourages users to verify, whenever possible, the integrity of downloaded software. For more information, see
http://www.cert.org/incident_notes/IN-2001-06.h
Employ egress filtering
Egress filtering manages the flow of traffic as it leaves a network under your administrative control.
In the case of the Trojan horse Sendmail distribution, employing egress filtering can help prevent systems on your network from connecting to the remote intruder-controlled system. Blocking outbound TCP connections to port 6667 from your network reduces the risk of internal compromised machines communicating with the remote system.
Build software as an unprivileged user
Sites are encouraged to build software from source code as an unprivileged, non-root user on the system. This can lessen the immediate impact of Trojan horse software. Compiling software that contains Trojan horses as the root user results in a compromise that is much more difficult to reliably recover from than if the Trojan horse is executed as a normal, unprivileged user on the system.
Recovering from a system compromise
If you believe a system under your administrative control has been compromised, please follow the steps outlined in
Steps for Recovering from a UNIX or NT System Compromise
Reporting
The CERT/CC is interested in receiving reports of this activity. If machines under your administrative control are compromised, please send mail to cert@cert.org with the following text included in the subject line: "[CERT#33376]".
Appendix A. - Vendor Information
This appendix contains information provided by vendors for this advisory. As vendors report new information to the CERT/CC, we will update this section and note the changes in our revision history. If a particular vendor is not listed below, we have not received their comments.
This is a good reason to compare the MD5 checksum of anything you download, source or binary, to what the author says it should be, especially if you downloaded from a mirror. Better yet, the author could use GnuPG and sign the code with his/her private key. Since only his/her public key can decrypt it, you know that the code has very likely not been tampered with.
bytesmythe
Hypocrisy is the resin that holds the plywood of society together.
-- Scott Meyer
Difficult? With GnuPG you can just do a gpg --verify and md5 is just md5sum filename
You mentioned the real issue: key management. If files from ftp.sendmail.org get infected, then people could probably get a bogus key as well. (Although the sendmail folks verified that the files didn't have an updated signature and wouldn't have validated).
Red Hat is good for this...use "rpm -K" to verify packages.
FreeBSD's ports system also automatically checks MD5 sums on downloaded files.
That being said, it certainly isn't inconceivable that the checksums themselves could be tampered with, but it is at the least a further layer of security.
----------
Floccinaucinihilipilification - the action or habit of judging something to be worthless
Yes, but if you read the full article, the source was also PGP signed, specifically so that tampering could be detected. Short of a miracle, it would be impossible for any cracker to fake a PGP signature, as they wouldn't have the required private key. So if you checked that (as it was specifically intended for), then you pick it up, and I expect that may quite possibly have been how it was detected.
Umm...yeah so some malicious code got through, but ummm....what about MS not releasing info on bugs until 3 months later, or how about them not being about to figure out why servers running 2k are being cracked(how politcally correct) left and right.
After reading the posting we'll note this is VERY similar to the OpenSSH trojan. The trojan doesn't wind up in the sendmail binary but is actually created during the build process.
So more than just checking the MD5sums of things you download you need to watch who you compile as, since the trojan will have the privledges of whoever compiled sendmail. This isn't exactly the most sly trojan either, it is quite blatent about how it creates a tunnel to a specified target, this can also help the intruder avoid firewall rules and detection.
If you find you've been affected by the trojan you would be wise to reinstall the system from known clean code since the intruder may have already created other backdoors from themself.
Gentoo Linux validates checksums on every package before unpacking; doing this pedantically prevented the BitchX trojan as well as many other things.
Just another plus for the distro, I suppose.
For end users of binary RPMs, RedCarpet (and I presume competing tools like apt-get) automatically check signatures.
Yes, it is more difficult for tarballs. But I think the theory is that people packaging tarballs have a clue. It really is not that difficult to check sendmail tarballs. Just download that ascii signature file also, and run pgpv. It even runs md5 for you. The only wrinkle is that you have to gunzip the tarball first:
Of course, it would certainlly be funny if all the modified binary did was, in addition to the normal functionality of the program, also told you that you should verify the relevant checksums before installing.
---
"Of course, that's just my opinion. I could be wrong." --Dennis Miller
But, no one can quite be sure that the binary is built from the official, non-trojaned source, even if they give the offical checksum for the distro.
Red Hat solves this problem by signing all their packages. It's up to Red Hat to assure that the sources aren't trojaned, but they have a big monetary motivation to do so.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
Ummm, I suppose the answer would be because they did figure it out several weeks ago.
Try keeping up with both sides of the news.
Yup, I got slimed, and I'm not an easy person to slime. This dude is the first person ever to get one up on me. But I'll have my revenge. I diffed the malicious source tree with the authentic one and found the malicious code. It looks amazingly innocuous until you base64 decode the shell script :-). His IP address is 66.37.138.99.
>What amazes me is I wasn't even aware anyone
>really used Sendmail anymore.
What amazed me is I wasn't even aware anyone really used Unix anymore. Man, look at all the security holes in *that* software's history.
Sendmail hasn't had a remote exploit in over two years. Named has had a single advisory posted against it (a DoS) since the 9.x series was released.
Matt
Packages like djbdns and qmail make service configuration trivial, as it should be.
Last time I checked you need to devote some serious time and brain power to properly setting up sendmail.
>After the number of open e-mail relays I've had to
>deal with, sendmail leaves a sour taste in my
>mouth.
Sendmail hasn't allowed relaying at all for about five years unless you explicitely turn it on. In otherwords, blame site admins, not Sendmail.
Matt
there was a single DoS one that i know of, but other than that no exchange 2000 exploits exist. i've never heard of a total remote access exploit for 5.5 or 2000.
outlook web access / iis is the only source of problems, but that's all iis' fault.
-f
www.blackant.net
I have made the backdoor'd sendmail code available at http://www.enzotech.net/files/sm.backdoor.patch and the base64 portion is decoded at http://www.enzotech.net/files/sm.backdoor.base64.t xt
This was diff'd from a previously downloaded tar ball that we were using for analysis of another bug.
>Packages like djbdns and qmail make service
/etc/mail/local-host-names, and commenting out "DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')".
/etc/mail/virtuserstable. Want to send all the mail in the domain? Change it to "@mydomain.net". Want to create dummy accounts named mydomain-USERNAME? Make the line "@mydomain.net mydomain-%1".
>configuration trivial, as it should be.
It really depends on what you're doing with them.
First off, installing either usually involves compiling from source, due to the restrictive licensing.
The software also isn't built to start under native Unix facilities like init or xinetd, so you either need to deal with installing and configuring ucspi and tcpserver, or hacking the source.
The lack of standards support in qmail also ends up requiring patching in a lot of common configurations, further complicating things. And honestly, configuring qmail isn't exactly a dream - permissions issues are easy to miss, a lot of configuration data requires compilation to a binary format, and configuration data is spread across a fairly large number of files and is hardly in the most intuitive of formats, e.g.:
+foo.com-:foo.com:500:500:/domains/foo.com:-::
Djbdns can either be very easy (simple case) or a pain in the ass.
For starters, there are a lot of standards it doesn't bother starting (IXFR, NOTIFY, TSIG, DNSSEC, etc). So interoperability with other servers is at best suboptimal.
Even if you're going to run a pure djbdns environment, there's no built in facility to transfer zone data, so you also need to either install and configure the axfr-dns program or install and configure rsync and ssh or some other method of copying the zone data to the remote server. In either case the propogation isn't automatic, so you'll probably want to write your own scripts to automate this.
Doing split-dns is a lot less intuitive than in BIND, in my opinion. You can add location tags to entries in the database, but the format only allows one per entry, so if you have different location definitions, you need to add the data multiple times (or store your data in a different format and process it into what tinydns-data expects). And those location tags are limited to only two characters, which prevents any meaningful name.
>Last time I checked you need to devote some
>serious time and brain power to properly setting
>up sendmail.
When did you last check? The mid-ninties?
Configuring Sendmail since the switch to m4 files is *trivial* for most setups.
What's even better is that, since Sendmail is under a permissive license, virtually all versions of Unix offer it by default, pre-installed and with a basic configuration.
For example, if you're doing a single domain setup the extent of your configuration on Red Hat would be adding your domain name to the
Want mail sent to phil@mydomain.net to go to bob instead?? Simple, add "phil@mydomain.net bob" to
And on Red Hat you don't even have to deal with compiling the database files, since the init scripts do it for you. Just run "/sbin/service sendmail restart".
Matt
I can't say for sure but seeing as the trojans only action is to open the port (doesn't infect anything, can't survive a reboot), it's probably not smart enough to cover it's tracks very well and that would probably show it.
The solution given in the cert advisory is basicly
"A language that doesn't affect the way you think about programming, is not worth knowing" - Alan Perlis
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
In theory. Not in practice. MD5, but more so SHA-1 is believed to be "collision free" -- it is computationaly infeasable to find 2 messages x and y where H(x) = H(y).
Finding not only these 2 messages, but a "malicious" code that would still preserve the original MD5 or SHA-1 is exponentially more infeasable, I would venture to say, impossible in the next few years. (Barring quantum computing)
(its much simpler to break into the main CVS server and just change the code there)
OK, I'll bite and start the religious war.
/etc/mail/local-host-names, and
/service/qmail-smtpd /service/qmail-smtpd
> The software also isn't built to start under native Unix
> facilities like init or xinetd, so you either need to deal with
> installing and configuring ucspi and tcpserver, or hacking
> the source.
Not true. It's simple to run qmail out of inetd; it simply isn't recommended because most implementations of inetd suck.
tinydns and dnscache do their own listening (since they require a persistent process and/or use UDP). axfrdns can also be run out of inetd, although the provided method for configuring it assumes daemontools/ucspi-tcp.
> a lot of [qmail] configuration data requires compilation to a
> binary format
Which is different from sendmail's aliases building how? Clearly some data requires fast lookup, and therefore some sort of advanced binary format. You can either have an editor which manipulates the binary format directly, or you can have an easy to read text format that gets converted to a binary format.
*ALL* of the alternatives discussed (qmail, djbdns, sendmail, and BIND) choose the latter approach.
> and configuration data is spread across a fairly large
> number of files
I (and DJB) consider this to be a feature. If you want to add a
an SMTP domain to receive mail for, would you rather type 'echo mydomain.com >>/var/qmail/control/rcpthosts' or would you rather write a tool to find the 'rcpthosts' section of the config file (which now needs extra text to denote each section) and add the entry there?
As another example, sendmail's config is moving toward this, but they're burdened by the legacy sendmail.cf file. Configuration now comes from many different m4 files. However, rather than have sendmail read them directly, it's "compiled" into the sendmail.cf (wasn't that also what you were complaining about previously?)
> and is hardly in the most intuitive of formats, e.g.:
> +foo.com-:foo.com:500:500:/domains/foo.com:-::
DJB file formats are typically designed to be easy to for programs to parse, create, or edit. They may not be "intuitive" but I find they are easy to make sense of, because they attempt to specify information in the format closest to the conceptual format. Config files are typically tables, with newlines delimiting each row. Some files (rcpthosts) have only a single column, while others use colon-delimited columns (tinydns/root/data). The only thing that isn't intuitive is that you need to know what each column is for (though this is easily specified in documentation).
And the best part is that it's simple to generate these formats, so if I want to implement special logic (say, for every domain in a list, point the A record for www.domain, their MX record, and their NS records at a particular IP), it's trivial to generate the format using existing tools.
> For example, if you're doing a single domain setup the
> extent of your configuration on Red Hat would be adding
> your domain name to the
> commenting out
> "DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1,
> Name=MTA')".
Qmail:
echo domain >>/var/qmail/control/rcpthosts
# don't run qmail-smtpd on 127.0.0.1! With daemontools,
# this would be:
cd
rm
svc -dx . log
There are similar responses for your other examples, but I won't belabor the point.