CERT: Sendmail Distribution Contained Trojan Horse

Scoria writes "According to a CERT advisory published this afternoon, the public distribution of Sendmail 8.12.6 contained a trojan horse from September 28 to October 6. For more detailed information, please consult advisory CA-2002-28." This sounds very much like what happened to OpenSSH.

Comments

But that's okay...

[Anonymous Coward]

As long as you could also get the source to the Trojan, as well... right?

Over a week?

[Anonymous Coward]

Many eyes = better security but only when many > 0

Re:Over a week?

[Wdomburg]

>What amazes me is I wasn't even aware anyone
>really used Sendmail anymore.

What amazed me is I wasn't even aware anyone really used Unix anymore. Man, look at all the security holes in *that* software's history.

Sendmail hasn't had a remote exploit in over two years. Named has had a single advisory posted against it (a DoS) since the 9.x series was released.

Matt

Re:Over a week?

[enneff]

Packages like djbdns and qmail make service configuration trivial, as it should be.

Last time I checked you need to devote some serious time and brain power to properly setting up sendmail.

Re:Over a week?

[Wdomburg]

>Packages like djbdns and qmail make service
>configuration trivial, as it should be.

It really depends on what you're doing with them.

First off, installing either usually involves compiling from source, due to the restrictive licensing.

The software also isn't built to start under native Unix facilities like init or xinetd, so you either need to deal with installing and configuring ucspi and tcpserver, or hacking the source.

The lack of standards support in qmail also ends up requiring patching in a lot of common configurations, further complicating things. And honestly, configuring qmail isn't exactly a dream - permissions issues are easy to miss, a lot of configuration data requires compilation to a binary format, and configuration data is spread across a fairly large number of files and is hardly in the most intuitive of formats, e.g.:

+foo.com-:foo.com:500:500:/domains/foo.com:-::

Djbdns can either be very easy (simple case) or a pain in the ass.

For starters, there are a lot of standards it doesn't bother starting (IXFR, NOTIFY, TSIG, DNSSEC, etc). So interoperability with other servers is at best suboptimal.

Even if you're going to run a pure djbdns environment, there's no built in facility to transfer zone data, so you also need to either install and configure the axfr-dns program or install and configure rsync and ssh or some other method of copying the zone data to the remote server. In either case the propogation isn't automatic, so you'll probably want to write your own scripts to automate this.

Doing split-dns is a lot less intuitive than in BIND, in my opinion. You can add location tags to entries in the database, but the format only allows one per entry, so if you have different location definitions, you need to add the data multiple times (or store your data in a different format and process it into what tinydns-data expects). And those location tags are limited to only two characters, which prevents any meaningful name.

>Last time I checked you need to devote some
>serious time and brain power to properly setting
>up sendmail.

When did you last check? The mid-ninties?

Configuring Sendmail since the switch to m4 files is *trivial* for most setups.

What's even better is that, since Sendmail is under a permissive license, virtually all versions of Unix offer it by default, pre-installed and with a basic configuration.

For example, if you're doing a single domain setup the extent of your configuration on Red Hat would be adding your domain name to the /etc/mail/local-host-names, and commenting out "DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')".

Want mail sent to phil@mydomain.net to go to bob instead?? Simple, add "phil@mydomain.net bob" to /etc/mail/virtuserstable. Want to send all the mail in the domain? Change it to "@mydomain.net". Want to create dummy accounts named mydomain-USERNAME? Make the line "@mydomain.net mydomain-%1".

And on Red Hat you don't even have to deal with compiling the database files, since the init scripts do it for you. Just run "/sbin/service sendmail restart".

Matt

Checksums

[SexyKellyOsbourne]

...and that's why you should actually use those MD5 checksums, instead of unpacking and installing without thinking.

Re:Checksums

[meatpopcicle]

Thats always a good idea, but I have a deeper question. How do these patches make it into the CVS of these projects and who is doing it?

I can think of some likely players, namely those who feel that Linux/Unix is a threat to them.

Also can't forget about the black hats and chinese/russian/terrorist groups as well.

Re:Checksums

[egg troll]

Egg Troll wants to know what if the trojaned program was the one with the checksums? MD5s, while an important tool, aren't a cure-all.

Re:Checksums

[Anonymous Coward]

Also can't forget about the black hats and chinese/russian/terrorist groups as well.

Incorrect md5 sums certainly strike terror into my heart.

Re:Checksums

[Henry V .009]

Do you know that the given MD5 was correct? If the got to the binary, what's to stop them from altering the MD5 given on the website.

Re:Checksums

[delta407]

Gentoo Linux validates checksums on every package before unpacking; doing this pedantically prevented the BitchX trojan as well as many other things.

Just another plus for the distro, I suppose.

Re:Checksums

[aridhol]

So don't use MD5. Use PGP or some other signature instead of checksums. The only problem then is getting authentic keys; that's where keyservers and the Web of Trust come in.

Re:Checksums

[Anonymous Coward]

That's why MD5SUM files are signed with the appropriate public key, so you can check the integrity of the file using gpg. Yes its a pain, but for security critical stuff its worth it.

What we need is a new format, as universal for Unix as .tar.gz is, which is signed so the decryption can only take place with the appropriate key installed or provided. Redhat might for instance ship a distro that recognizes keys from itself, as well as sendmail, openssh, mozilla, etc. to make unpacking these signed archives automatic. If you grab a signed archive from me, you'd have to provide the decryption software with my public key to unpack it. You don't need to use it for everything, just security critical stuff. We have this in our browsers to protect end users, but we allow backdoors in through the...er...back door due to this oversight.

Re:Checksums

[AntiFreeze]

apt-get also validates MD5 checksums before installing a package. Very helpful, but of course, if someone can modify the binary that is getting distributed, what would stop them from modifying the checksum?

Of course, it would certainlly be funny if all the modified binary did was, in addition to the normal functionality of the program, also told you that you should verify the relevant checksums before installing.

Re:Checksums

[delta407]

apt-get also validates MD5 checksums before installing a package.

Yeah, sure, you can validate md5sums on binaries. But, no one can quite be sure that the binary is built from the official, non-trojaned source, even if they give the offical checksum for the distro.

Gentoo neatly gets around this problem by using the source directly, and since a lot of projects list md5sums of the source archives (such as sendmail 8.12.6), Portage can make sure that it gets the correct tarball.

Oh, and by the way:

$ cat /usr/portage/net-mail/sendmail/files/digest-sendma il-8.12.6
MD5 73e18ea78b2386b774963c8472cbd309 sendmail.8.12.6.tar.gz 1867436

So, Gentoo had the right one on file all along. And, of course, Portage won't unpack files with the wrong md5sum, meaning Gentoo users were completely immune to this.

Re:Checksums

[GigsVT]

But, no one can quite be sure that the binary is built from the official, non-trojaned source, even if they give the offical checksum for the distro.

Red Hat solves this problem by signing all their packages. It's up to Red Hat to assure that the sources aren't trojaned, but they have a big monetary motivation to do so.

Re:Checksums

[zeekiorage]

...and that's why you should actually use those MD5 checksums, instead of unpacking and installing without thinking.

Although I agree that you should check the MD5SUMs of all the software you download, the advisory says that the sendmail FTP server was compromised, the intruders could have easily uploaded new MD5 checksums along with the source code, I am surprised they didn't.

Re:Checksums

[kasperd]

How do these patches make it into the CVS

Where did you read that it was in CVS? I didn't read that anywhere. According to the article the trojan was only in the compressed tar files available for download.

Re:Checksums

[Tim Browse]

Yeah, sure, you can validate md5sums on binaries. But, no one can quite be sure that the binary is built from the official, non-trojaned source, even if they give the offical checksum for the distro.

Huh? Isn't that kind of the point of using md5?

If you have an md5 checksum for a binary (and assuming that it's from a reliable source), then why can't you use this to validate the binary is correct?

You could, in theory, construct a trojaned-binary that had the same md5 checksum, but I had always thought that this was so difficult as to be infeasible/not worth worrying about.

What am I missing? Are you saying the md5 checksum is being spoofed too?

Tim

Re:Checksums

[conway]

You could, in theory, construct a trojaned-binary that had the same md5 checksum, but I had always thought that this was so difficult as to be infeasible/not worth worrying about.

In theory. Not in practice. MD5, but more so SHA-1 is believed to be "collision free" -- it is computationaly infeasable to find 2 messages x and y where H(x) = H(y).
Finding not only these 2 messages, but a "malicious" code that would still preserve the original MD5 or SHA-1 is exponentially more infeasable, I would venture to say, impossible in the next few years. (Barring quantum computing)
(its much simpler to break into the main CVS server and just change the code there)

Re:Checksums

[Yokaze]

Um... don't know about RedHat, but Mandrake RPMS are signed. When you want to install packages from a different source, say, like Penguin Liberation Front, you'll get some warning about failing to verify the packages-signature.
To get rid of the warning and to verify the source, you'll have to add the public key of PLF to your root's keyring.

Re:Checksums

[rasjani]

Im not sure how gpg key works in rpm's but i would like to think that Noone except accept to redhat's private key could sign them the package. Thus, in conjunction with checksums, its the best choise to avoid stuff like this to happen.

Re:Checksums

[BlueUnderwear]

(its much simpler to break into the main CVS server and just change the code there)

... or break into the ftp server, and change the "reference" MD5 sum too.

Re:Checksums

[jeremyp]

And what if it was an inside job? If it was a disgruntled employee extracting revenge, they obviously didn't bother to do anything with the PGP sigs either because they didn't have access to the private key or because they knew that nobody ever bothers to check the signature when they download a sendmail or any other distro. But supposing it was an inside job and they did change the signatures or even worse inserted the trojan into the source CVS tree....

FYI Sendmail Inc did have to lay some people off recently which seems a bit of a coincidence to me.

Re:Checksums

[Tassach]

RPM has integral cryptographic signing and checksumming. Any RPM based distro takes care of this automagically. I'm not an APT user, but I suspect it has similar features.

Re:Checksums

[delta407]

What am I missing? Are you saying the md5 checksum is being spoofed too?

What I'm saying is this. Even if the binary matches the binary your distro gives you, nothing proves the distro got clean sources. Yes, they can check the md5sum on their source tarball, but that's another window for error.

Gentoo goes for the sources directly, and verifies they are correct before doing anything with them.

Re:Checksums

[Tim Browse]

Yes, they can check the md5sum on their source tarball, but that's another window for error.

Is there a better way of making sure you have the correct source files?

Gentoo goes for the sources directly, and verifies they are correct before doing anything with them.

How? By visual inspection? Code review?

Tim

Re:Checksums

[Blkdeath]

Although I agree that you should check the MD5SUMs of all the software you download, the advisory says that the sendmail FTP server was compromised, the intruders could have easily uploaded new MD5 checksums along with the source code, I am surprised they didn't.

The MD5 Sums are kept on the website as well as in a signed file on the FTP server. If they altered the signed file, the signature would return an error. Besides that, the intruders would also have to update the website to alter that MD5 Sum.

That's a lot of work, and crackers don't seem up to the task, it seems.

Re:Checksums

[delta407]

Yes, verified by md5sums of the source tarballs. Basically, my point boils down to the fact that binary distros may or may not check their sources at build time.

We need a way to verify signatures

[lamj]

PGP signing is a good way to prevent trojaned software like this case. But I think the process to verify the software is too complicated and not easy for all users to use. Let me ask you this, when is the last time you checked the hash or PGP signature after you download a software?

For most people, never.... It would be great if we have automatic download tools to check signature as well (obviously, we need standard for storing the signature as well)

Re:We need a way to verify signatures

[quitcherbitchen]

Difficult? With GnuPG you can just do a gpg --verify and md5 is just md5sum filename

You mentioned the real issue: key management. If files from ftp.sendmail.org get infected, then people could probably get a bogus key as well. (Although the sendmail folks verified that the files didn't have an updated signature and wouldn't have validated).

Re:We need a way to verify signatures

[spencerogden]

Source Mage (The distro formerly known as Sorcerer) checks MD5 sums on download if they are known. I don't have sendmail installed, so I don't know if it would have caught this one. I think so because the MD5 sum on file for sendmail was created on Aug. 20.

Re:We need a way to verify signatures

[ubernostrum]

Red Hat is good for this...use "rpm -K" to verify packages.

Re:We need a way to verify signatures

[taion]

FreeBSD's ports system also automatically checks MD5 sums on downloaded files.

That being said, it certainly isn't inconceivable that the checksums themselves could be tampered with, but it is at the least a further layer of security.

Re:We need a way to verify signatures

[tuffy]

If files from ftp.sendmail.org get infected, then people could probably get a bogus key as well.

This difference, though, is that one can download a public GPG key from a site (like sendmail.org or something) and continue using it to verify software over several versions. So, for example, you could use a 2002 public GPG key to verify software in 2004 and be reasonably sure the key hasn't been tampered with for two years straight without someone noticing. With an md5sum, the checksum is only good for that version of the software and can forged much easier in the short term.

That said, I think md5sums are better for ensuring integrity, GPGs are better for ensuring security and both should be as automated as possible (like with the help of RPM and friends).

Re:We need a way to verify signatures

[CustomDesigned]

rpm -K sendmail-8.12.6-1.src.rpm

For end users of binary RPMs, RedCarpet (and I presume competing tools like apt-get) automatically check signatures.

Yes, it is more difficult for tarballs. But I think the theory is that people packaging tarballs have a clue. It really is not that difficult to check sendmail tarballs. Just download that ascii signature file also, and run pgpv. It even runs md5 for you. The only wrinkle is that you have to gunzip the tarball first:

$ pkg=sendmail.8.12.6.tar
$ gunzip <$pkg.gz >$pkg
$ pgpv -v $pkg.sig
This signature applies to another message
File to check signature against [sendmail.8.12.6.tar]:
Good signature made 2002-04-05 19:37 GMT by key:
1024 bits, Key ID 678C0A03, Created 2001-12-18
"Sendmail Signing Key/2002 <sendmail@Sendmail.ORG>"

$ rm $pkg

Re:We need a way to verify signatures

[ishark]

PGP signing is a good way to prevent trojaned software like this case. But I think the process to verify the software is too complicated and not easy for all users to use. Let me ask you this, when is the last time you checked the hash or PGP signature after you download a software?

Always. It goes like:
- download (mirror) the updates for my distro (Mandrake)
- rpm --checksig *.rpm

The Mandrake key is added to root's keyring off the CDROM at install time, so unless someone hacks your box to change it, it will be the right one.

Re:We need a way to verify signatures

[bogado]

Not being pedantic, but MD5 is diferent from a dignature, a signature is much more safe. It relies in a secret key that only the developer holds, this key is used to encript a MD5 sum. Only the public key from the original packager can decript the MD5 correctly. This allows you to be certain that the person who packaged the software is the holder of the secret key.

RPM suports MD5 and signatures, but not all packages out there are actualy signed, the one from the major distros (red hat, ximian, mandrake) are certainly signed.

Re:We need a way to verify signatures

[Ded Bob]

The MD5 checksums are distributed from a FreeBSD server independent of the source distribution server. They would have to change the MD5 in two locations to make that work.

Article text... (before it gets /.ed)

[whatisthatvelvet]

CERT® Advisory CA-2002-28 Trojan Horse Sendmail Distribution
Original release date: October 08, 2002
Last revised: --
Source: CERT/CC

A complete revision history is at the end of this file.

Overview
The CERT/CC has received confirmation that some copies of the source code for the Sendmail package were modified by an intruder to contain a Trojan horse.

Sites that employ, redistribute, or mirror the Sendmail package should immediately verify the integrity of their distribution.

I. Description
The CERT/CC has received confirmation that some copies of the source code for the Sendmail package have been modified by an intruder to contain a Trojan horse.

The following files were modified to include the malicious code:

sendmail.8.12.6.tar.Z
sendmail.8.12.6.tar.gz

These files began to appear in downloads from the FTP server ftp.sendmail.org on or around September 28, 2002. The Sendmail development team disabled the compromised FTP server on October 6, 2002 at approximately 22:15 PDT. It does not appear that copies downloaded via HTTP contained the Trojan horse; however, the CERT/CC encourages users who may have downloaded the source code via HTTP during this time period to take the steps outlined in the Solution section as a precautionary measure.

The Trojan horse versions of Sendmail contain malicious code that is run during the process of building the software. This code forks a process that connects to a fixed remote server on 6667/tcp. This forked process allows the intruder to open a shell running in the context of the user who built the Sendmail software. There is no evidence that the process is persistent after a reboot of the compromised system. However, a subsequent build of the Trojan horse Sendmail package will re-establish the backdoor process.

II. Impact
An intruder operating from the remote address specified in the malicious code can gain unauthorized remote access to any host that compiled a version of Sendmail from this Trojan horse version of the source code. The level of access would be that of the user who compiled the source code.

It is important to understand that the compromise is to the system that is used to build the Sendmail software and not to the systems that run the Sendmail daemon. Because the compromised system creates a tunnel to the intruder-controlled system, the intruder may have a path through network access controls.

III. Solution
Obtain an authentic version of Sendmail
The primary distribution site for Sendmail is

http://www.sendmail.org/
Sites that mirror the Sendmail source code are encouraged to verify the integrity of their sources.

Verify software authenticity
We strongly encourage sites that recently downloaded a copy of the Sendmail distribution to verify the authenticity of their distribution, regardless of where it was obtained. Furthermore, we encourage users to inspect any and all software that may have been downloaded from the compromised site. Note that it is not sufficient to rely on the timestamps or sizes of the file when trying to determine whether or not you have a copy of the Trojan horse version.

Verify PGP signatures
The Sendmail source distribution is cryptographically signed with the following PGP key:

pub 1024R/678C0A03 2001-12-18 Sendmail Signing Key/2002
Key fingerprint = 7B 02 F4 AA FC C0 22 DA 47 3E 2A 9A 9B 35 22 45
The Trojan horse copy did not include an updated PGP signature, so attempts to verify its integrity would have failed. The sendmail.org staff has verified that the Trojan horse copies did indeed fail PGP signature checks.

Verify MD5 checksums
In the absence of PGP, you can use the following MD5 checksums to verify the integrity of your Sendmail source code distribution:

Correct versions:

73e18ea78b2386b774963c8472cbd309 sendmail.8.12.6.tar.gz
cebe3fa43731b315908f44889d 9d2137 sendmail.8.12.6.tar.Z
8b9c78122044f4e4744fc447eea fef34 sendmail.8.12.6.tar.sig

As a matter of good security practice, the CERT/CC encourages users to verify, whenever possible, the integrity of downloaded software. For more information, see

http://www.cert.org/incident_notes/IN-2001-06.ht ml
Employ egress filtering
Egress filtering manages the flow of traffic as it leaves a network under your administrative control.

In the case of the Trojan horse Sendmail distribution, employing egress filtering can help prevent systems on your network from connecting to the remote intruder-controlled system. Blocking outbound TCP connections to port 6667 from your network reduces the risk of internal compromised machines communicating with the remote system.

Build software as an unprivileged user
Sites are encouraged to build software from source code as an unprivileged, non-root user on the system. This can lessen the immediate impact of Trojan horse software. Compiling software that contains Trojan horses as the root user results in a compromise that is much more difficult to reliably recover from than if the Trojan horse is executed as a normal, unprivileged user on the system.

Recovering from a system compromise
If you believe a system under your administrative control has been compromised, please follow the steps outlined in

Steps for Recovering from a UNIX or NT System Compromise
Reporting
The CERT/CC is interested in receiving reports of this activity. If machines under your administrative control are compromised, please send mail to cert@cert.org with the following text included in the subject line: "[CERT#33376]".

Appendix A. - Vendor Information
This appendix contains information provided by vendors for this advisory. As vendors report new information to the CERT/CC, we will update this section and note the changes in our revision history. If a particular vendor is not listed below, we have not received their comments.

Checksums and crypto

[bytesmythe]

This is a good reason to compare the MD5 checksum of anything you download, source or binary, to what the author says it should be, especially if you downloaded from a mirror. Better yet, the author could use GnuPG and sign the code with his/her private key. Since only his/her public key can decrypt it, you know that the code has very likely not been tampered with.

Re:Checksums and crypto

[spencerogden]

If no one would notice if the md5 sum changed, would they notice if someone changed the public key?

Re:Checksums and crypto

[AntiFreeze]

I'm not sure I completely understand what you're saying. Here's what I propose (not certain if it's what you're saying or not):

The author of a package should digitally sign the package he is distributing, and part of that signature should necessarily contain the appropriate MD5 checksum for the package.

This way, one can easily tell if the package has been tampered with (if the signature does not unencode with the author's public key, or the unencoded checksum is invalid). Of course, the one flaw with this is that there needs to be a way to determine who the author is and grab the author's public key. If the author's identity is stored in the package, then this method is useless. If the author's identity can be found via some central resource (say, a searchable database of projects and their authors on the FSF or GNU websites), then the security of this method is only as strong as the security of that database.

All in all, food for thought, but no real foolproof way of verifying you're installing untampered binaries.

Scary.

[QuantumWeasel]

It's been a long time since I installed sendmail or inn or bind from sources. At some point I stopped checking MD5 signatures, and now I trust the major distros to do that for me. I sure hope they're more vigilant than me. And I used to be so paranoid... This is a nasty wake-up call.

Not to plug Gentoo AGAIN, but...

[bytesmythe]

Gentoo automagically checks MD5 sums when it downloads source packages during installs/updates. Very handy.

Re:Not to plug Gentoo AGAIN, but...

[delta407]

If I emerged sendmail 4 days ago... have I been trojaned?

Ahem.

$ cat /usr/portage/net-mail/sendmail/files/digest-sendma il-8.12.6
MD5 73e18ea78b2386b774963c8472cbd309 sendmail.8.12.6.tar.gz 1867436

That's the same md5sum as the correct one listed in the CERT advisory, and that's the md5sum Portage looks for before proceeding to unpack. So, no, Portage won't let you be trojaned.

check sums blah blah

[domninus.DDR]

everyone says just check sums, but how are these people changing the file? If they can change the tarball on the server than why not change the page to have thier md5?

Re:check sums blah blah

[quitcherbitchen]

It would be best to sign the MD5 with a PGP signature. They key they use may have also been compromised, but at least that adds another layer to security.

Re:check sums blah blah

[iamplasma]

Yes, but if you read the full article, the source was also PGP signed, specifically so that tampering could be detected. Short of a miracle, it would be impossible for any cracker to fake a PGP signature, as they wouldn't have the required private key. So if you checked that (as it was specifically intended for), then you pick it up, and I expect that may quite possibly have been how it was detected.

Re:check sums blah blah

[mosch]

Well, the solution to that is a distributed checksum, such as is found in the FreeBSD ports system.

Type cd /usr/ports/mail/sendmail && make install and it then downloads the source, and then checks what it downloaded against an md5 checksum that's kept on your machine before it applies patches and builds.

Re:check sums blah blah

[Jucius Maximus]

"everyone says just check sums, but how are these people changing the file? If they can change the tarball on the server than why not change the page to have thier md5?"

You get the binaries from a mirror to ease the load on the master server and then get the MD5sum (which is just a few bytes) from the master server to verify the binaries you got.

Microsoft Sux!!!

[Anonymous Coward]

What?! It's not M$? oh.......

Thank GOD for Microsoft!

[eamber]

Good thing I use Exchange Server. I've got a tight ship there.

Re:Thank GOD for Microsoft!

[Sabalon]

Don't forget that according to the earlier article you will now need to pay extra for that tight ship - otherwise you get the submarine with the screen door.

This is a good reason to get windows!

[greenskyx]

That way when you get your software you know who put the security holes in it. It's all part of trustworthy computing... ;-)

Sendmail

[Anonymous Coward]

Further proof that security through obscurity don't work.

Only the FTP...

[OneFix]

According to the advisory, it was only the FTP site that was compromised (The HTTP was fine).

So, as for those that are saying it's an Open Source problem, this is just wrong.

There's been alot more closed software distributed with Viri/Trojan Horses. The truth is, this is bound to happen if the public archives are on an unsecured server...I even seem to remember pressed CDs being distributed with trojans.

So, what are they doing to keep this from happening again?

Re:Only the FTP...

[Quixote]

I even seem to remember pressed CDs being distributed with trojans.

Surely these can't be Microsoft CDs!?! According to a KB article at Microsoft.com, "Disks are duplicated on a variety of industrial strength, quality focused systems. Most of these systems are UNIX-based. The UNIX-based duplication systems used in manufacturing are impervious to MS-DOS-based, Windows-based, and Macintosh-based viruses."

Re:Only the FTP...

[duffbeer703]

Actually, GA releases of two Microsoft products for non-english languages shipped with the Klez worm.

Re:Only the FTP...

[OneFix]

So, lets get this right...you're trying to blame a UNIX machine for a Mac/Windoze virus???

All of the virus scanning should be done by the vendor/distributor...

The infections generally happen before it gets pressed. That's why it's usually only a few files that are infected. Someone's machine is infected and due to either poor administration or what not, they get onto the pressed CD.

But, the truth is, mass-produced CDs don't go into a CD-R (ever wonder why there's no dye to be found on pressed CDs?)...They are pressed...they use molded metal "stamps" from a glass master...

The UNIX machines are most likely only running the pressing machines...now, if you're expecting me to belive that a virus can get onto the pressing machine through this process, I'ld like to know how...

Check out This link to read about the process of CD pressing.

I'm sure M$ has a reason for making it sound like they are using standard CD-Rs for this process...it probably makes it easier to blame a UNIX machine when a problem does arise...rather than telling ppl that one of your developers had an infected machine...

Re:Only the FTP...

[Elwood P Dowd]

Surely these can't be Microsoft CDs!?! According to a KB article at Microsoft.com, "Disks are duplicated on a variety of industrial strength, quality focused systems. Most of these systems are UNIX-based. The UNIX-based duplication systems used in manufacturing are impervious to MS-DOS-based, Windows-based, and Macintosh-based viruses."

Um. I can't tell if you're kidding or not, so I'll bite:

It doesn't matter what kind of computer runs the machine that copies the CDs. The machine that creates the master CD could have a virus, and infect an executable on the CD. I'm sure microsoft has a number of failsafes in order to make sure that this master CD doesn't have a virus on it, but having a unix computer run the duplication machine is not one of those failsafes.

Re:Only the FTP...

[Xtifr]

So, lets get this right...you're trying to blame a UNIX machine for a Mac/Windoze virus???

Just the opposite -- he's saying that there can't be any viruses on the MS CDs because MS uses Unix, not Windows, to press their CDs. And of course, that's silly, but the quote from the MS site is (or was) real -- it made the rounds of many Unix/Linux mailing lists, and I went to the site and saw it myself. They've probably removed the quote (and maybe even switched OSes for their pressing machines) after all the publicity it got, but it was definitely there at one time.

(MS's point was that there can't be any viruses introduced during the pressing process because their Unix machines were immune to viruses. Of course, this is irrelevant and it doesn't help with pre-existing viruses on the images, but MS was trying to reassure their customers, not confuse them with facts.)

Two reasons.

[Komrade S.]

1) They obviously didn't have access to the website, as the HTTP downloads of Sendmail were NOT compromised
2) It didn't get discovered for a week, so obviously no one used the check sums.

Re:We knew...

[robson]

We always knew it was a piece of insecure crap. so no surprise. Qmail or even mailx is MUCH more secure.

(Except this wasn't due to any vulnerability within the software itself -- it looks like it was a lapse in the security of the sendmail project's server.)

Question

[Door-opening Fascist]

Is doing a

# netstat -a | grep 6667

all that is necessary to see if one has a the open port, or is there more to it than that?

Re:Question

[jericho4.0]

Moderators!! This is on topic!

I can't say for sure but seeing as the trojans only action is to open the port (doesn't infect anything, can't survive a reboot), it's probably not smart enough to cover it's tracks very well and that would probably show it.

The solution given in the cert advisory is basicly

1. 1. reboot

1. 2. install untrojaned sendmail.

Holy crap

[Anonymous Coward]

I actually did a postfix install during that period.

I considered doing sendmail, but then I remembered how fucking thick the ORA sendmail book is and how complex it is, so I decided, "screw it, let's try postfix, I have never tried it before." If I had gone with sendmail, there would be some serious egg on my face tomorrow morning. We might be running MS exchange within a week if that had happened.

(Oh yeah, and postfix was pretty easy to set up.)

Re:this is why you cannot trust OSS

[The Analog Kid]

Umm...yeah so some malicious code got through, but ummm....what about MS not releasing info on bugs until 3 months later, or how about them not being about to figure out why servers running 2k are being cracked(how politcally correct) left and right.

Upgrading Sendmail isn't Enough

[fliplap]

After reading the posting we'll note this is VERY similar to the OpenSSH trojan. The trojan doesn't wind up in the sendmail binary but is actually created during the build process.

So more than just checking the MD5sums of things you download you need to watch who you compile as, since the trojan will have the privledges of whoever compiled sendmail. This isn't exactly the most sly trojan either, it is quite blatent about how it creates a tunnel to a specified target, this can also help the intruder avoid firewall rules and detection.

If you find you've been affected by the trojan you would be wise to reinstall the system from known clean code since the intruder may have already created other backdoors from themself.

No worries!

[Greyfox]

Microsoft'll have a secure version out (For a small fee) ANY DAY NOW!

Ahem. Sorry. Couldn't resist. AAH! Don't mark it troll yet! Keep reading!

Ok, folks will say "Well here's a great example of a problem cryptography would prevent." Well as long as the guys inserting the trojans aren't contributing to your code base. Minor detail there. Keep in mind that a "trojan" can be as easy to code up as allowing a buffer overflow to take place (AND you have plausible deniability there.) Ok. So I'm paranoid.

So lets talk about the crypto side of things again. Since I'm paranoid and all that. Do you trust the project maintainer's system security? Reckon he allows anyone to log into his system? Do you trust their security and the network they come over? For that matter, reckon the CVS archive the code's stored on could be compromised? Do you see what we're up against yet? Paranoia...

Ok, lets say we've checked out his sytem and it's sterling. Key server/key management is a big pain in the ass right now. It'd be nice to have some infrastructure in place where I could go to a brick and mortar, establish my identity (Here's my passport, driver's license yadda yadda) and load MY PGP public key onto their server with their signature attached. Might even be worth a few bucks for me. That'd make that whole expiration thing pretty easy to deal with too.

It also seems to be that the US Postal Service would be the ideal venue for this infrastructure. As much of a pain in the ass as they are to deal with, it'd make the whole key revocation/renewal thing much easier. And it'd be a whole lot more secure than me asking my friends to sign my key via E-mail.

Re:No worries!

[Greyfox]

Kinda like VeriSign, yeah. Except with identity verification. And for my PGP key. That's why the Brick and Mortar bit is so important. If I could just put my PGP public key on a disk or CD and take it down to the nearest post office, show my driver's license and social security card (or passport) and have them sign my key right there and load it up to their public key server that would rock. And if any citizen could do that for $10 or $20 (maybe per year) you might even start to have an infrastructure that the public would be willing to use.

Re:No worries!

[Greyfox]

Oh yeah! I don't want to cast aspersions as to the integrity of the open source development model. At least with them I can look at the source code. Since I can't look at closed source programs, I really can't be sure that some three letter agency might not have dictated a back door in there (did I mention I was paranoid, by the way?) I really don't have any way to prove otherwise. Leastwise with open source I can always be sure there isn't one, simply by booting my (openbios controlled) computer, firing up my hand-coded assembler (copy con assembler.com), compiling my hand assembled C compiler and bootstrapping GCC with it... Did I mention I was paranoid by the way?

Re:No worries!

[CableModemSniper]

It also seems to be that the US Postal Service would be the ideal venue for this infrastructure. As much of a pain in the ass as they are to deal with, it'd make the whole key revocation/renewal thing much easier. And it'd be a whole lot more secure than me asking my friends to sign my key via E-mail.

Right up until there I was with you. The Post Office? That's the last place I'd want with my PGP/GPG key. Because of course, as soon as a service like this was offered people would jump at the chance to use it. It could quite quickly and easily replace Passport and Project Liberty, etc. But don;t you think the government would say to themselves, oh look...we let people associate public keys with real people reliably. Let's make sure it's really reliable, and require them to give us their private key as well when they sign up. I mean they have been looking for a way to get escrow keys for a looong time, this would give them the perfect excuse.

Now who would I trust to provide this service? That's a good question. You can't really trust the government because its yet another way for them to track you. You wouldn't want to entrust it to a bank, who knows what they would do with that information. But there needs to be a place that can verify your identity and associate it with a public key. Conceivibly you could see firms whose express purpose is to provide this service. Of course this means confusion and possible incompatibilty when the person's key you want to verify and your key are under two different verification companies.

The only possible solution I can see is the classical web of trust. But again that has its own problems.

What we really need is some kind of incorruptable agency for handling this sort of thing. Of course we all know what the chance of something like that showing up is...

Re:A Sad Day for Egg Troll

[benwb]

While sendmail is much more secure now, in the days of yore it made IIS look like fort knox.

Does any one know

[gorjusborg]

How long does it take for all the trojan infested code to propagate out of use?

I wonder how many admins download/install packages, go on vacation (missing all the warnings), and simply never hear of the problem.

I think it would be interesting to see some statistics on this topic.

Hardly news ...

[jc42]

Let's see, a Trojan Horse is basically defined as an undocumented chunk of code hiding inside a program, which does something that you don't know about or understand.

Sendmail is such a complex beast that, no matter how much you personally know about it, there are always things in there that you don't know about or understand.

So it has always been full or Trojan Horses.

This is the fundamental thing that's wrong with building a hugs program that tries to do everything possible. Pretty much all the other mail tools are better at sendmail in this respect, because they only try to be a mail tool.

Sendmail, OTOH, is an emulator for a rather complex sort of machine language. Some time back, someone demonstrated that it was possible to emulate a Turing machine with a sendmail.cf file. Impressive as this may be technically, it's way overkill for the task, and it shouldn't be any surprise to anyone when problems turn up in sendmail and aren't discoverted for a while.

It's guaranteed that there are others lurking inside that monster.

Re:Hardly news ...

[jc42]

> By that definition, most linux software is a trojan

Well, yeah; I often tell people that all the programs that I write contain THs. For example, I usually include a number of debug hooks that jack up the verbosity. Most users don't want to know about these things, so I only document then down at the end of the man page, where it won't be seen. So to most users, my programs can suddenly start writing huge log files that they weren't expecting.

I also often include code that isn't quite working yet, and only enabled via some command-line flag that isn't documented. I can tell a few select beta testers about it. To everyone else, such things are TH code.

People do write a lot about such things without ever making their definitions clear. It can be fun to make up a "reasonable" definition and see if you can argue that it applies to your own stuff.

Re:Hardly news ...

[Trogre]

Let's see, a Trojan Horse is basically defined as an undocumented chunk of code hiding inside a program, which does something that you don't know about or understand.

Not quite.
A Trojan Horse is defined as a big wooden horse which sat outside the ancient city of Troy, just large enough to happily contain 700 greeks in full battle dress and still leave adequate room for toilet facilities.

For more information read Homers's Iliad.

Re:Hardly news ...

[Saint Aardvark]

Dear God, you're right; I thought you were joking.

http://www.hopf.demon.co.uk/demon/turing.txt

Re:Hardly news ...

[jc42]

I thought you were joking.

As usual, when I joke, people respond as if I were serious, and when I'm serious, people think it's a joke.

But at least someone took the time to Look It Up ...

Re:Upgrade?

[FyRE666]

That's why I'm still running sendmail from 1996!

Could you let us know the IP address so we can all put that "swiss cheese" server in our blacklists? ;-)

Ah...

[The Bungi]

Is this like that script worm that was shipped with the .NET framework in Taiwan or India or something like that earlier this year?

Hmmm. Except this looks a bit uglier.

Of course, the headline for that was very different. A bit more, let's say, sensationalistic. Yeah, that's the word I was looking for.

Re:Ah...

[alyandon]

This is a different issue altogether as anyone building sendmail from the contaminated source would end up with an infected system.

The script worm you are referring to was found in an html help file which was a native language translation done by a third party and inadequately checked by MS before mastering the cd. You couldn't get infected by installing .NET from the cd and the particular exploit the script relied on was fairly old -- so any recent version of IE wouldn't have executed the script.

It just goes to show that there is no such thing as 100% secure. I can be really bad about not checking signatures on source I download before compiling/installing.

So how do I fix it?

[alehman]

How 'bout a quick tutorial from someone who knows pgp or gpg or MD5 on how to use it to figure out if my recent install is trojaned?

Also, the CERT advisory doesn't give any fixes, it just gives the signatures. It doesn't seem like installing a good version would eliminate the trojan.

Re:So how do I fix it?

[bogie]

Oh no, cause that would actually be helpful. Funny how yours is the most relevant question and yet a bunch of stupid MS jokes are modded higher.

I would mod you higher, but since I modded down Alan Cox I've never had mod points again. He was offtopic though.

*yawn*

[comet_11]

It seems like every time a new trojan/worm/misc virus hits the scene, a thousand posts go up accusing that software of being horribly insecure and advocating some other software of the same type as being better.

It's quite simple... software can be infected by viruses, open source, closed source, any operating system or language. Just because today it's Sendmail that took a hit doesn't mean that it couldn't be qmail tomorrow.

If you got a virus, don't blame it on the software you downloaded, blame it on yourself for not validating it first.

Simply buy the book that's available.

[BoomerSooner]

I did. It is Life with QMail on steroids. Plus it has a great troubleshooting section. And every part goes further into detail. The only thing out of date is the daemontools but if you can read the info that you download with it you'll be fine.

Buy the book, you'll thank me later. QMail is great but it's a very detailed process of installing all the associated s/w. relay-ctrl is critical as well (to avoid becoming a spam server for "Smoking your way thin" and "Get confident stupid!" seminars (ahhh I miss Troy McClure).

OU 17 TX 10

Huge applications are bad.

[miffo.swe]

Big apps like sendmail and exchange that wants to do everything are always going to be bug ridden. Thats mainly because the code is so big that its impossible to have a complete picture of how everything interacts. Installing and hiding trojans in a large CVS tree is easier too than in a small tree. Learn from MS on this one. Do everything halfbad or one thing excellent.

Putting software on secure servers or better to put them on a community server would be nice. I can understand that coders dont want to tinker on their servers when they want to code but why not get some help?

Ximian Red Carpet

[JoeBuck]

Ximian's Red Carpet does MD5 and GPG verification of packages every time it does an install or an update.

qmail anyone? :)

[portege00]

I realize that qmail wouldn't solve the problem of modified tarballs that allow trojans to come alive during builds (that's what md5s are for), but if you're really worried about security you'd probably be using qmail anyway. If you can prove me, the author, and everyone else who has a qmail fetish wrong, there's a prize in it for you.

After the number of open e-mail relays I've had to deal with, sendmail leaves a sour taste in my mouth. Using the blacklist that has no real regulation on it doesn't seem to help, either. Closing a relay makes users upset. Sendmail is a lose-lose situation, and now there's a trojan in it to top it off. Wee!

Re:qmail anyone? :)

[mbogosian]

I've preferred qmail for quite some time, but I don't use it at home because it's a pain to install (relative to just tweaking my existing sendmail setup which I've grown to understand). I'd love it see it move into more major acceptance, but the problem is that sendmail is the default for many distros. RedHat doesn't even have a qmail RPM to install.

Re:qmail anyone? :)

[Wdomburg]

>After the number of open e-mail relays I've had to
>deal with, sendmail leaves a sour taste in my
>mouth.

Sendmail hasn't allowed relaying at all for about five years unless you explicitely turn it on. In otherwords, blame site admins, not Sendmail.

Matt

Re:qmail anyone? :)

[cymen]

If you're running linux you could check out qinstall:

obua.org

They have a script for FreeBSD too but I prefer:

Matt Simerson's Qmail Toaster

I've got a couple sites running Qmail, djbdns (tinydns, dnscache, etc), courier, and sqwebmail/squirrelmail and it simply rocks. It is a bit of pain to get installed but the actual configuration of the software itself is so simple it is worth the pain. Bind vs tindydns config files are simply hilarious because the tindydns one is just so damn simple! Qmail is much the same.

Re:qmail anyone? :)

[cymen]

Matt Simerson's Qmail Toaster

Doh...

Re:qmail anyone? :)

[Wdomburg]

>I'd love it see it move into more major
>acceptance, but the problem is that sendmail is
>the default for many distros. RedHat doesn't even
>have a qmail RPM to install.

You won't see that happen unless Bernstein changes the license. Mandrake, Red Hat, Debian, and the OpenBSD maintainers have all stated explicitely that they cannot ship qmail because of the license.

The issue is even more prevalent now that distributions are striving for LSB compliance, as every piece of Bernstein's software flouts the FHS, and even traditional Unix conventions (BINARIES DO NOT BELONG IN VAR).

Even if they chose to ignore that and added qmail in a Bernstein approved way, they'd be taking a huge step backwards in functionality, since the license also doesn't approve of shipping modified source or binaries, and qmail has extremely limited standards support. No more out of the box support for SSL/TLS connections or secure authentication. They'd have to instruct their customers to download the source, patch it themselves, and recompile if they wanted that.

Qmail's lack of adoption lies solely on Bernstein's shoulders, noone else's.

Matt

Re:A Sad Day for Egg Troll

[benwb]

Yes, of course, that was exactly what I meant. Thank goodness you were on the ball, or someone would have thought that I was alluding to Fort Knox's traditional reputation of extremely high security.

Who do you trust?

[Gerry Gleason]

So lets talk about the crypto side of things again. Since I'm paranoid and all that. Do you trust the project maintainer's system security? Reckon he allows anyone to log into his system? Do you trust their security and the network they come over? For that matter, reckon the CVS archive the code's stored on could be compromised? Do you see what we're up against yet? Paranoia...

It isn't that complex to do it right, but as we all know, it's the human factors that get you. I'm sure GnuPG has all the functions necessary to implement it, but you need a trusted party with rock solid proceedures to ensure the top of the chain. What do people do for a CA for their signatures?

As long as you can be certain that: 1) You have the correct public key for the signing authority, and 2) nobody but the signing authority can get access to the corresponding private key. You can do it yourself by generating a key set for your own CA to a floppy and then making signing keys for yourself and your friends from there. You only need the public key to make a certificate, so a friend can email it to you, you get out your floppy and sign with GnuPG, and send the cert back. Keeping your signing keys off-line is a good idea too (if your paranoid, but who can afford not to be with this stuff going on).

Now, the only point that can be attacked is to compromise the CA's signing certificate (This is the CA's public key, signed by itself). If you squirl away a copy of it and get a new copy from time to time to double check you should be completely safe. You could use a public CA, but the commercial ones tend to charge a lot, so it would be nice to have a cooperative CA that does it on the cheap, but still does it well (Does such a thing exist?). Since someone is obviously getting their jollies by compromising distributions on public ftp servers, I'd be a little careful about setting this up. As long as the root signing key is safe (This is the private key), you can make lots of copies of the root certificate (on differenct servers, of course) and verify them with each other periodically.

With all of this in place it should be a simple matter to script the verification of signed signatures. I know I'm not the only person who knows how to do this correctly, so perhaps this is already done. If not, it looks like an excellent project for someone wanting to do this stuff.

Re:Who do you trust?

[crucini]

It's possible to sketch out a very convincing protective scheme based on crypto, but it becomes more shaky when host based weaknesses are taken into account. This is what I think Greyfox was talking about:

Developer Dave is working on program P on his workstation W. A while back, he gave an account to his friend Frank. Frank unwittingly logged in from a compromised host, and now the attacker Adam has a shell on W. Adam escalates to root by exploiting some weakness in a SUID program and ensures his access with a hacked sshd.

Adam knows from reading Dave's mail that Dave is getting ready for a release of P. Dave builds the tarball, fetches the floppy with his secret key from secure storage, and signs the tarball. But in the interval Adam has replaced the tarball with a modified one. Dave signs the compromised tarball and releases it to the world.

Re:Who do you trust?

[Gerry Gleason]

Adam knows from reading Dave's mail that Dave is getting ready for a release of P. Dave builds the tarball, fetches the floppy with his secret key from secure storage, and signs the tarball. But in the interval Adam has replaced the tarball with a modified one. Dave signs the compromised tarball and releases it to the world.

As I pointed out, the people processes are the weak point. You can make a compromise arbitrarily hard, and you can fix things for the future if there is a compromise. All I was doing is pointing to the important points in the process. The window you describe is very small, and it can be closed if you are that paranoid. Adam can't do that if you take your computer off the network when you do the signing. If Adam has rooted one or more key machines of an important developer, he could add his trojan to the code base (CVS repository) which is a lot easier than trying to compromise the tarball before it is signed.

An approach short of taking your machine off the network is to work on a machine that doesn't take any incoming connections. This is easier than you might think. I'm behind a simple and cheap NAT box, and I don't run any services that require incoming ports to be openned. Even if I had an MS machine and used Outlook, it would only be possible to infect that machine. A sophisticated virus would be able to reach through the firewall to establish a connection, but it would only be to that machine.

The people who count know all of this, and they are careful about the important stuff. The point is to limit damage. All I am saying is that you have to be particularly careful about dealing with the root keys that are used in implementing the Certificate Authority, becuase if those are compromised, someone could forge completely authentic looking signing keys, and it would be very hard to detect and fix. Even then, you revoke the CA keys and make new ones, and this time be more careful (paranoid). If you are working with the root keys, you had better be absolutely sure that host based weaknesses are eliminated.

Even Verisign could be compromised, but it isn't likely because their business depends on people trusting that this can never happen. I've done this kind of work in a commercial setting, so I know what the issues are. That's the whole point of the question in my subject, you have to trust someone, but it doesn't have to cost you a forture (I've recently seen $400 as a price for a Verisign key).

Re:Who do you trust?

[NearlyHeadless]

Adam knows from reading Dave's mail that Dave is getting ready for a release of P. Dave builds the tarball, fetches the floppy with his secret key from secure storage, and signs the tarball. But in the interval Adam has replaced the tarball with a modified one. Dave signs the compromised tarball and releases it to the world.

The Trusted Computing/Palladium key management stuff could help a lot to prevent/detect compromised hosts.

Although everyone talks about the DRM implications of Palladium, there are other uses. The voting machines in Brazil, for example, could use the system to make sure the votes reported are signed by a chain of software that goes all the way to the hardware. There's no way to ensure that in software only.

Re:Who do you trust?

[crucini]

As I pointed out, the people processes are the weak point.

Even though I used a human "error" as the gateway to the hypothetical hack, such a hack could occur with little or no human error. The real lesson is that general purpose computers cannot be fully trusted. There's just too much going on between Tempest emissions, mysterious firmware in lots of peripherals, very complex modern operating systems with sometimes unpredictable faults. All usable OS's are crumbling piles of buffer overflows waiting to be exploited.

Remember, when DES was standardized NIST would not approve software implementations. It's generally believed that NIST was representing the NSA. Since then computers have not gotten simpler or more reliable.

I'm behind a simple and cheap NAT box, and I don't run any services that require incoming ports to be openned.

Someone commented a few days ago that since the exploits are being squeezed out of the common daemons the attackers must eventually shift to client-side code which has received little scrutiny so far. Any estimates of how many exploitable buffer overflows are present in Mozilla? This will be a new era, because computers natted behind firewalls are typically much less secure than computers on the internet.

Even Verisign could be compromised, but it isn't likely because their business depends on people trusting that this can never happen. I've done this kind of work in a commercial setting, so I know what the issues are.

Verisign's not likely to be compromised by immediate-gratification kiddies who would harm their reputation. Although in fact this may have happened - if attackers stole card numbers by MITMing transactions with a fake Verisign certificate, it was probably never discovered. Someone who runs an e-commerce site told me that 25% of his transactions are fraudulent, with stolen card numbers. In other words, while abuse of a card number is obvious, it is not obvious where the card number came from.

Anyhow, if there are more dedicated groups who really want to compromise Verisign, they may well have done so. I'm afraid you will bridle at this comment, having experience in this area. Since I know nothing of Verisign's particular security arrangements, I'll just mention some general considerations:

1. Security by routine gets rusty unless constantly probed by an adversary. The Air Force used to send phony relief crews to Titan missile bases to see if they could talk their way into the control center. Does Verisign continously attack its security arrangements with well-funded, independent, skilled adversaries who are thoroughly briefed on the security arrangements? I suspect a commercial organization finds it more profitable not to.
   
2. Does Verisign monitor the personal lives of employees with access to cryptographic material? Many people in the valley are feeling the pinch of recession. If an employee can't make the mortgage payment due to a laid-off spouse, he could be pressured into selling out.
   
3. How many copies of their private keys does Verisign keep? I assume that the primary copy is in tamperproof hardware used for signing. But there must be backups or the business is quite vulnerable to a bit changing in an EPROM. How are the backups stored, and what procedure is used to safely transport the backup data to the production equipment when necessary? Is this procedure drilled, examined, attacked?
   
4. How does Verisign handle an audit, such as a BSA audit? How about fire inspection? Insurance inspection? What if a fire occurs inside the building? Won't they have to evacuate and give the fire department free reign?
   
5. Under the USA PATRIOT Act, the FBI can demand Verisign's keys and they cannot tell anyone about it. Has it happened yet? If so, how did they authenticate the FBI agents?
   
6. Would Verisign ever admit to being compromised?
   

All of these comments point to the same idea: serious cryptographic security is not possible for private commercial entities.

Re:Who do you trust?

[Gerry Gleason]

All of these comments point to the same idea: serious cryptographic security is not possible for private commercial entities.

Yes, it is. You seem to be talking about perfect security, which is probably never possible because of the nature of negative proofs. My claim is that you can get as high a level as you need/want if you pay attention to the right details. I'm not going to deal with all of your points because you don't really seem to be concerned with real problems, just poking holes without any reference to the actual situation at hand.

The only issue here is to be able to make sure that you can rely on the digital signatures used in releasing and distributing the code. Yes, there are other places to compromise the system, but those are much less likely. They are still analysing what happened in this case, but the code base was never compromised, just the download site. This would have been immediately detected if everyone checked their downloaded tarballs against the signatures.

Given the existence of this attack, it is necessary to go a bit further to clean up some problems, and the key is to make a minumal robust process easy so there is no reason not to do it. That is exactly the problem I was addressing, so a lot of this is completely irrelevant. If signatures are verified back to a trusted CA an attack of this type would be next to impossible.

Because the CA is the key point, I will address a couple of your points related to that. No, if they design the process correctly, there are no complete key copies anywhere. We used split keys stored on crypto-cards, password protected and stored in a subdivided safe. Two people were needed to do anything so you would have to have a conspiricy to break the system. If we were evacuated, the fire department would have to use their axes to get into the computer room, and we would immediately cancel any certificates that could logically be compromised. I'm sure that Verisign has proceedures just as rigorous.

Even the FBI could not get our keys. We were doing this before the export of strong encryption was allowed, but we got around that because all messages went through our hub where they were encrypted under keys we controlled (as outlined above). The BSA export license specified what we were responsible for if presented with a warrent. All we were responsible for was unwrapping the strong crypto outer wrapper, and giving back the unwrapped message. This would be done in a Tempest room with no network connections of any kind, and when completed the keys went back in the safe. I don't think we would have been required to let the government person in the room for the process, just give them the unwrapped message.

Would anyone admit if the process was compromised? There would be a lot of pressure not to, but it would likely leak out. Lots of people at the company would know and might leak it, particularly if they were fired or badly treated.

Like the subject says, it all comes down to who do you trust, and well run companies that treat their people well will be the most trustworthy. Networks of dedicated volunteers are even more trustworthy, if not quite as reliable. The transparency of the process is critical to establishing the networks of trust in this case. It is exactly the opposite of security through obscurity, and that's what makes it the most robust in the long run.

I got the bastard's IP

[archnerd]

Yup, I got slimed, and I'm not an easy person to slime. This dude is the first person ever to get one up on me. But I'll have my revenge. I diffed the malicious source tree with the authentic one and found the malicious code. It looks amazingly innocuous until you base64 decode the shell script :-). His IP address is 66.37.138.99.

Re:I got the bastard's IP

[mbogosian]

Yeah, this guy looks pretty mean. Incidentally, unless it's been recently changed, 66.37.138.99 points to spatula.aclue.com:

Decisionism Inc. (ACLUE2-DOM)
4260 E. Evans Ave.
Denver, CO 80222
US

Domain Name: ACLUE.COM

Administrative Contact, Technical Contact:
Klein, Eli (NCMGTMOSXI) elijah@firstlink.com

4260 E. Evans Ave.
Denver, CO 80222
US
no phone no fax

Record expires on 17-Dec-2003.
Record created on 17-Dec-1998.
Database last updated on 8-Oct-2002 23:09:51 EDT.

Domain servers in listed order:

NS3.FIRSTLINK.COM 66.37.141.4
DENVER.FIRSTLINK.COM 66.37.143.67

Any chance this isn't the guy responsible (i.e., Eli had his machine h4x0r3d)? Talk about an ironic choice of domain names.... At least he's running Apache.

Re:I got the bastard's IP

[boolie]

oops. thought i was logged in.

previous post is from eli, yes, this is the owner of 66.37.138.99 *gasp*, and more importantly, no i'm not involved in the backdooring of sendmail in any way.

Re:How the heck?

[Ziviyr]

Yeah, its nice we don't have anything that like makes a focused beam of ions or anything.

Noone will be putting trojans on my chips!

Re:Sendmail is now worthless, instead

[Wdomburg]

>Well, If you don't mind using a proprietary
>program written by a person with the same level of
>maturity and social skills as a mentally retarded
>5 year-old.

Don't forget "that hasn't had a new release in over four years and neglects to implement a lot of basic standards like authentication or encryption".

Matt

Re:LMAO!

[Wdomburg]

>It is still funny, simply because it is yet
>another sendmail problem.

Yeah, and if someone breaks into your house and pees on your carpet, it's yet another carpet problem.

Matt

got any exploits?

[honold]

there was a single DoS one that i know of, but other than that no exchange 2000 exploits exist. i've never heard of a total remote access exploit for 5.5 or 2000.

outlook web access / iis is the only source of problems, but that's all iis' fault.

Stop Supporting MD5 Checksums!!

[Anenga]

MD5 Checksums have a higher rate of collisions, both in the wild and artifically. A machine can be built for only around $100k or less which can find collisions in less than 24 hours. Hell, in a few years standard computers could probably generate collisions easily. SHA1 (Simple Hash Algorithm) is a much better alternative over MD5.

The previous version of MD5, MD4, was so flawed it is now considered "broken". "Dobbertin [Dob95] has shown how collisions for the full version of MD4 can be found in under a minute on a typical PC... Clearly, MD4 should now be considered broken.".

SHA1, while of the same family of hashes as MD4 and MD5, remains uncompromised by any research discoveries, and is widely used in many applications requiring the highest levels of security.

Gnutella, the File Sharing Protocol, uses SHA1 over MD5 for the same reasons I state here. A developer of Bitzi (the Metadata/Hash catalog) has also recommended to the Gnutella Developer Forum not to use MD5, but SHA1 instead. Thus, people should be using SHA1 instead of MD5. I've noticed some major websites and companies are using MD5 hash's now, such as Adobe and Roxio. I would recommend to them to change them to SHA1 instead, since Gnutella supports it (and the fact that it is a much more secure and stronger hash algorithm)... and they can use MAGNET URI's to link to the files on Gnutella.

Read and realize what's going on before you post!

[MavEtJu]

(sorry, I have to get this out of my system)

READ THE ARTICLE AND REALIZE WHAT IS GOING ON!

It says that:
The FTP-server of sendmail.org was compromised.

It doesn't say that:
- somebody commited code to the CVS server.
- nobody reads the commitlog of the CVS server.

It says that:
The sendmail-distribution was trojaned.

It doesn't say that:
- sendmail itself was trojaned
- there are trojans inside sendmail
- qmail/postfix is better because it isn't trojaned.
- exchange is better because the source is closed. It's the distribution which is corrupted, not the software.

It says that:
The correct MD5-checksum is ...

It doesn't say that:
- with PGP signing it wouldn't be prevented. Security is a process, you need to follow the rules or you are not secure. You should check all checksum/signatures you have, preferable from independant resources (e.g. one from sendmail.com and one from your unix-distribution).

Next time, please read the article and realize what's going on before you post (apologies to the people who actually did :-)

Edwin (yes, the guy from the OpenSSH trojan)

OpenBSD not affected:

[friscolr]

for those of you who read /. instead of the mailing lists for your OpenBSD news, OpenBSD is not affected

From: Todd C. Miller
To: mr grip
Cc: misc@
Subject: Re: sendmail trojan - were stable or current affected?

In message so spake "mr grip" (jhonold):

> does anybody know if either tree in the last couple months had trojaned code
> in it, exploiting make builders?

Not affected. When I committed sendmail 8.12.6 the tarball I fetched
was not the trojaned one. Even if it had been we probably would
not have been affected since we don't use sendmail's build system
(which is where the trojan was apparently inserted).

- todd

The backdoor code

[netmask]

I have made the backdoor'd sendmail code available at http://www.enzotech.net/files/sm.backdoor.patch and the base64 portion is decoded at http://www.enzotech.net/files/sm.backdoor.base64.t xt
This was diff'd from a previously downloaded tar ball that we were using for analysis of another bug.

Re:The backdoor code

[MavEtJu]

Heh, at least they left out the crap-talk about Crays and HP-UXs this time :-)

Re:LMAO!

[cscx]

That carpet... it really tied the room together, man.
</ lebowski quote>

Re:LMAO!

[Wdomburg]

I was wondering if anyone would think the same thing I was when I was writing that comment. :)

Matt

Detecting corruption vs. tampering

[billstewart]

MD5 checksums are good for detecting corruption - bad downloads, etc. They don't help against tampering, because the Bad Guy can also tamper with the checksum, unless there's some separate distribution method. That's why public-key signatures are so important.

Maybe an FTP server security hole?

[billstewart]

FTP servers have a reputation for bad implementations, though perhaps I'm just annoyed at having had a wu-ftp crack let my machine get hosed a year or two ago. If it wasn't an inside job, then it's likely to have been some exploit like that - which is especially likely, since the HTTP server's version was clean.

Why Mail Shouldn't Run As Root!

[billstewart]

Sigh. Yet another set of exploits that are far worse because people run their mail systems as root. It's just not necessary - the System V side of the world hasn't done this in decades. Sendmail comes with a bunch of mechanisms to make it run as a user most of the time, but that just means you need to exploit it fast and hard :-) Here are some of the reasons people run mail systems as root

• Too much embedded base to fix. Yes, that sendmail. There are competing mailers - postfix, qmail, smtpd, etc.
  
• Delivering mail to users' mailboxes. The System V world handles this with group permissions, by making the mailboxes group mail and running the mail delivery system as group mail instead of user root. Works fine.
  
• Port 25 "secure" because root-only. The naive implementation is to run the smtp daemon as root to access it; the other naive implementation is to use inetd or its relatives to open the port but run the application as a user. A much better solution would be to modify the kernel to do a least-privilege approach, letting some other approved user own the port, though that's got obvious portability issues for applications that want to run on multiple flavors of Unix. But it ought to be done.
  
• Running user-provided filters on incoming mail, e.g. procmail or vacation-mail senders. This is a bit harder - it's easy enough to implement a mail-user vacation-mail daemon, but doing a procmail that doesn't need to be root but doesn't let arbitrary users abuse it takes a bit of work; the filter could easily require group root privileges to run, and only really needs to acquire the recipient's userid if it's accessing private directories - that's inherently unsafe (because a bad procmail script could let people attack you), but if you want to do it, it shouldn't be too hard to set the filter to r-sr-x--- jruser mail or something like that. Better to have the user set up directories that group-mail can write to.
  

Re:Why Mail Shouldn't Run As Root!

[netmask]

The trojan doesn't work like that, look at the code. It compiles and runs when you run "./Build". It doesn't matter what sendmail runs.

THe moral of the story is, Check the SIG files.. AND don't configure/compile software as ROOT. At least then it would have only been user level compromise.

Scorched earth policy on bad distros?

[thogard]

I think sendmail.org should up the version number at once and kill the .6 version once and for all. That would allow many people to look at what they have and say "yep, its .6, throw it out" but they want to keep the old version number so people get to play games. There are many reasons why they won't have the origianl tar ball and they have a very simple way to insure people don't have the trojaned version.

Re:Scorched earth policy on bad distros?

[thogard]

My guess (based on professional orginazations) that there are about 50,000 competent sysadmins out there. That leave the rest below that level and if their boxes get rooted, we all pay the price. Its not like upping a version number is going to cost sendmail anything and it will remove the doubt of a trojaned version. Real sysadmins won't care when they read that the new version was increased because of something that won't concern them or there servers but it will help out the masses of pretend sysadmins.

You still need a trusted info source ...

[bockman]

Let's see ...
I download the tarball and MD5s. Then I want to verify the signature. For that I need a public key or something like that of the developer that signed the tarball.Since I never met him, I must resort to download also that from an internet place, probably the same from which I downloaded the source.

Now, what prevents whoever cracked the server and placed the troianed tarballs on it, to also change the public key, so that it matches the couterfeit signed tarball?

At a minimum, one should go to some forum/ML and check the key with a dozen or so other users, choosing the ones that got the key in different places and times.

Or am I missing something ?

Re:You still need a trusted info source ...

[TrentC]

I download the tarball and MD5s. Then I want to verify the signature. For that I need a public key or something like that of the developer that signed the tarball.Since I never met him, I must resort to download also that from an internet place, probably the same from which I downloaded the source.

There's where your error in understanding occurs. Public keys should never be received through the same channel as the files to be verified. If you are, then something is very wrong.

Many people who use them make them available through an email autoresponder, on a separate website, or through a public keyserver. Some distros ship them on their CDs.

I'll agree that there's still a certain amount of paranoia involved ("How do we know the box Red Hat generates its master CD image from wasn't cracked?") but at some point you have to trust the sender or the delivery machanism. If not, you might as well yank the net connection from your computer...

Jay (=

Re:this is why you cannot trust OSS

[Scoria]

Are you certain? :)

Must start to do back-traces

[Taco Cowboy]

Someone ought to start doing back-trace and see how that trojan horse get inserted in the package.

This "insert trojan horse" practice is becoming serious now. First it was OpenSSH, now it's Sendmail.

If we are not careful enough, next time we may see distros coming up with trojan horses all over the place.

Just like a chain will only be as strong as its weakest link, security will only be as good as where it breaks down.

Back-tracing is a must, in situation like this !

Re:Must start to do back-traces

[Neon Spiral Injector]

It is almost surely a comprimized machine itself, but it had to be logged into from somewhere (probally another comprimized machine).

Re:Sendmail is now worthless, instead

[Wdomburg]

>You must be use to MS who has to do contsant
>releases for bug fixes and more money. See,
>sometimes software can actually become stable.

Actually, no, the last time I ran an MS product at home for any length of time was DOS 5.0. I used NT 4.0 at one of my jobs for a period, but that was only to run a 3270 emulator for accessing the mainframe.

For the record, I *do* use qmail in a production environment, for thousands of domains. It requires a lot of patching to make it a usable product.

>Would you rather use a stable mail server that has
>really no known bugs in like 3-4yrs, or one that
>is contantly updated and has had a trojan horse?

Actually, I'd rather use a mail server that implements at least SOME of the standards that have come out in the past seven or eight years.

Stable doesn't mean jack shit if the software doesn't do what you need it to.

Matt

Re:Faulty Logic RE: MD5 Checksums and Integrity

[sqlrob]

Let's see - File stored on the server is changed

So, what makes the MD5 checksums stored right next to it proof from change?

Solution: Don't get the checksums from the same place you download the file.

Re:A Sad Day for Egg Troll

[jeremyp]

It wasn't sendmail, it was a worm that used a sendmail exploit to propagate. Only DEC VAXes running a particular Unix (4.2 BSD I think, but I could be wrong) were affected, admittedly that was a large proportion of computers on the Internet at the time.

Ironically, the worm was not written with malicious intent, but a bug caused it to go crazy.

Conspiracy Theorist

[Abreu]

Give it up, Mulder... Not everyone is out to get you

also keys come from DIFFERENT locations than src

[FreeUser]

If files from ftp.sendmail.org get infected, then people could probably get a bogus key as well.

This difference, though, is that one can download a public GPG key from a site (like sendmail.org or something) and continue using it to verify software over several versions.

Not only that, but public keys (or even complete keyrings containing public keys for groups of developers) can be obtained from multiple, different sources, all of which in turn are different and ideally independent of where one downloads the source tarball from.

This means one can obtain a developer's key or keyring from, say, a public key server (or two, or several), some ftp site (preferably a different non-mirror one from the tarball), a purchased CD, or any number of other places, check them against each other (make sure none disagree), and use them to check a download immediately, as well as 5 years from now.

The cracker would have to not only trojan the tarball, but also break into numerous independent key servers around the globe, numerous ftp sites around the globe, likely numerous web sites as well, and perhaps even various freenet nodes as well (if that is being used to distribute keys as well). And for those who anti up $5 for a CD with developers keys on it, they'd have to intercept the postal service and swap CDs as well (or crack the master CD before it goes to press).

Good luck. Even the NSA would probably have trouble pulling something like that off.

Re:Faulty Logic RE: MD5 Checksums and Integrity

[Tassach]

That's why the MD5 file is signed with GPG/PGP: to guarantee that it isn't tampered with. Of course, that requires that the user actually CHECK the signature.

Re:Faulty Logic RE: MD5 Checksums and Integrity

[sqlrob]

Would most even notice if it was replaced with a file that wasn't signed?

Let's Paaaaarty!!!!!

[KjetilK]

Well, you're right of course.

However, people would probably not fetch the public key from the same place, but from a keyserver. That helps a bit, but not a lot. There is nothing that stops anyone from generating a keypair with the same name and address as the original source, and quite possibly, you would see the file verified, but still by the wrong key.

Even if you would have two public keys, you would have no idea which one to trust. So, signing is no magic bullet.

We have to get much better at organizing keysigning parties. Keysigning parties are really the key (yeah, pun intended, puns are cool! :-) ) to this problem.

Gather friends every now and then, make sure you have a printout on you fingerprint on you at all times, whenever you travel, try to find somebody to meet, and so on. That way, we will build a web of trust which is so strong, people can identify valid keys, and then we're an awful lot better prepared to face this kind of problems.

Make sure you sign up at Biglumber, that you check it when you're out travelling, and that you subscribe to the "keysignings" list to keep up to date when things are happening in you area.

And, then, I'd really like to see my distro check sigs by default.

Yeah, and my keyid is 6A6A0BBC.

OK, I take it back!

[Tim Ward]

You do (sometimes) publicise non-MS vulnerabilities.