Slashdot Mirror
Known-Good MD5 Database
bgp4 writes "Have you ever examined a system you thought was broken into but you weren't sure? If only you had run an integrity verification program like osiris or Tripwire first you could have figured out what programs had been changed. In an effort to help out in the instances when you can't answer the question "what was this like before?" we've constructed a searchable database of MD5 and SHA-1 hashes for files in many standard operating systems. You can search using the filename or the checksum and see if you have a trojaned binary or an overactive imagination. Currently at knowngoods.org we have many FreeBSD, OS X, Linux, and Solaris installations checksummed and cataloged. If you have other programs or distributions you would like to see in the database, please let us know."
Wouldn't this be useless to anybody that builds from source?
This is the type of thing that you'd ask "Why didn't they do this sooner?" -- it's just that logical of an idea.
Absolutely fabulous, wonderful! The real trick, though, is to build up trust in your database so that those searching it will be sure that the checksums are actually correct--you know, rather than buying a burglar alarm from the robber himself. Thus, I doubt you'd be able to take submissions from users right away--at least without a competent staff checking to make sure they're correct.
I've been wondering when something along these lines would be available.
[devil's advocate] However, how do we know that the pregenerated checksums are correct? Who watches the watchers? [/devil's advocate]
Yah, yah, I know, the easiest way is to inspect the source for the minicompiler, the main compiler, and the program by hand, then build all of them step-by-step until you're done, then use the final binary to generate your hash. I wonder, tho, how much drift might there be in using a pre-built compiler (say I D/Led the binaries for GCC and the libraries to go with it). One tiny change in machine state (or any other number of things I would suppose) could result in the final binary being a single byte off, and the whole thing's a wash.
Granted, I may be talking out of my ass here, could someone w/ some hard-core coding knowledge or CS experience expound on the above?
Considering its history of vulnerabilities, I'd think that this would be pretty important...
What they don't say and what a lot of security folks forget to do is that they can't check your checksums of binaries on the same box. You need to copy the files to another box and check the checksums there with a known good version of your checksumming binary. The local version of your checksumming binary could have been compromised.
Heck, the utilities you used to pull the binary off the machine in question could have been compromised and may not be actually copying the binary in question, but a good version of the binary. The only way to check this would be to mount the drive on another machine and check it there... And if people aren't doing that (which it's a pain in the ass) all this website is going to do is give people a false sense of security.
All editorial writers ever do is come down from the hill after the battle is over and shoot the wounded.
A few months ago I put together a list of the "polymorphic" files in FreeBSD 4.6:
These files are always going to set off alarms if you've upgraded-by-source. (On the other hand, if a file *not* on this list has a different checksum, it probably just means that you've applied a security patch.)
Tarsnap: Online backups for the truly paranoid
Then I imagine that as soon as someone changes a hash, many secure systems will indicate they've been comprimised, and the whole thing will be quite obvious to sort out.
Boot from a known good floppy or CD to check your md5sums.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
Mu corporate www proxy filters this site as category "Hacking".
This is great for precompiled binaries, but it won't work so well for config files - they're different from system to system. I have a better solution:
/etc/passwd and /etc/shadow are especially likely to be modified, so I'd recommend sending those right away.
Anyone who wants to make sure their important config files haven't been changed by an intruder can email them to me, and I'll hold on to them for safe keeping.
Oh good, the md5 hash for my /sbin/md5 binary matches the signature found on known-goods. Now I can sleep at night. oh, wait...
_______
2B1ASK1
The fancy way to do that is with an Authenticode-like system for signing files. Distro maintainers would sign the files in their distros, and users could also sign their own files. A simpler way would be to just have a big, signed list of md5's in some file that tripwire checks against. Tripwire would check the signature on the file before believing the md5's in it. Or the list could contain individual signatures per file instead of just hashes.
A centralized md5 database doesn't feel so right with the free software spirit, which says (legitimate) users could modify the files at any time, or just recompile them with a slightly different compiler, etc.
Debian has this built into the OS with debsums.
It does require a legit dpkg database (and md5sum, and the debsums program itself...) but it's a nice tool.
I'd rather see everyone using bitzi.com, since it's
goal is to gather metadata for *every* file in the
universe, and keep the data free, supported by a
related business model (and a viable, sustainable
support mechanism is GOOD), but I support this
project too, because choice and freedom are goods.
Therefore, I urge everyone to submit metadata
to both projects.
If you only submit to one, however, please submit
to bitzi, because it provides an automation API,
and uses better hashes.
Note that I have no affiliation with the Bitzi company.
-I like my women like I like my tea: green-
NIST (The National Institute of Standards and Technology) currently has a program to provide this service, though largely focused on Microsoft OSes and associated apps. It may be found here: National Software Reference Library
The complete list of software they've checksummed can be found here: Software Listing or you can use their search engine if you're looking for a specific application here: Search Engine
Ideally, a simple tool should be developed that does the following:
Compare the MD5sums of critical files to a recent known "snapshot" of the system on RO media, which only indexes files that were changed and reconciled. Perhaps there is a list of files of which only certain byte ranges (perhaps just executable ELF sections) are checked, are some are omitted. (Other slashdotters mention caches/timestamps in certain relevant files that screw up checksums). You would have a whitelist (files which must match), then a graylist (files which meet byte-range criteria), and perhaps even a blacklist that prevents files that would normally be flagged to be ignored.
In checking full file checksums, those not explicitly listed above would fallback to a check using a HTTP get request conforming to this helpful document these guys have offered.
And to those who were asking about other distributions: they are looking for people willing to work with them to add new distros/architectures to their database.
Fuck Beta. Fuck Dice
NIST does this too. For a different reason though. To help forensic examiners eliminate non-important data in a suspect's computer. They use 4 different hash algorithms (MD5, SHA-1, CRC32, and one other), so good luck finding a collision for all 4. They were giving out copies of the CD-hashdb at an InfoSec conference I was at recently.
I'd also mention that it appears to be useless for BSD or Gentoo-like systems as well. BSD because it's built form source and the fingerprints won't always match, and Gentoo because there's already something like this built directly into the system, at least for verifying source tarballs.
Gentoo checks the md5sum of each tarball against another file containg the known value every time it installs something. The md5sums and the sources are obtained from different servers, so a lot of the risk of trojans is removed. Granted, this doesn't do continuous monitoring like this does, but it helps ensure you don't install something bad. The biggest worry now with this system could be vulnerable if several mirrors are hacked. They're working to replace it with a private-key signed system, which is much better than and md5 based system. The reason being that, that you can verify _who_ created the checksum in addition to that the checksum matches the file.
So, I'm not sure what the real benefit of this system is. It seems to be duplicating a lot of features that really should be built into the package manager ideally. Maybe someday we'll have package managers that actually watch their packages in realtime w/ strong crypto to make sure things are still good. That would be very cool.
Now I can add a compromised md5sum to my rootkit which uses values from this site.
Go team!
The reality of the matter is that, while it certainly would be possible for somebody to gag a machine to evade all your wascally checksumming tricks, they frequently don't do so. And when they do it, there's the usual arms-race lag between the time when a new method of checking comes out and when they update their tools to evade it. And there's a cost to them for each defense they evade; if you want to avoid every defense you ever hear of, you basically have to roll your own rootkits, which is a huge time investment.
And a kiddie who's out there collecting hundreds of boxes has no particular incentive to be anal about holding onto yours.
Fucking pompous amateurs.
I need a daemon that will automatically checksum the daemon. And then a daemon to automatically checksum the checksumming daemon. And a daemon to automatically checksum the daemon checksumming daemon checksumming daemon and a daemon...
Why not fork?
For Red Hat-based systems, at least, rpm -V will do pretty much exactly what you're looking for.
... The (mnemonically emBoldened) character denotes failure of the corresponding --verify test:
/etc/php.ini
/etc/httpd/conf/httpd.conf /var/www/html/index.html /var/www/html/poweredby.png
:)
From the man page for rpm:
The general form of an rpm verify command is
rpm {-V|--verify} [select-options] [--nodeps] [--nofiles] [--nomd5] [--noscripts]
Verifying a package compares information about the installed files in the package with
information about the files taken from the package metadata stored in the rpm database. Among other things, verifying compares the size, MD5 sum, permissions, type, owner and group of each file. Any discrepencies are displayed.
S file Size differs
M Mode differs (includes permissions and file type)
5 MD5 sum differs
D Device major/minor number mis-match
L readLink(2) path mis-match
U User ownership differs
G Group ownership differs
T mTime differs
So while that's a bit cryptic, a shell script run once every x days (30? 14?) should tell you what files have changed. All you would have to do is run rpm -qa to grab a list of the packages in your system, and then loop through the list and run rpm -V for each RPM returned.
For instance, running rpm -V on two common packages, Apache and PHP, shows me the following:
# rpm -V php
S.5....T c
(php.ini has changed... which in this case means I've tweaked some of PHP's default settings.)
# rpm -V apache
S.5....T c
missing
missing
(Okay, I've changed httpd.conf, again pretty much a given, and I've removed a couple of the default files.)
I guess this website seems pretty unneeded to me. Granted, the above is just for RPM-based systems, but I'm sure Debian and ports have similar options. And to the people who have installed from source and say "What about me?", I say, first, never underestimate the power of a package management system, and second, check out CheckInstall, which allows you to create an RPM or DEB just by saying "checkinstall" instead of "make install". If you feel you must compile from source, checkinstall is a necessity! Using checkinstall gives you all the benefits of a package management system while still allowing for the flexibility that compiling from source provides.
Between checkinstall and up2date, I'm a very happy Red Hat customer. I just wish more people knew about some of the truly powerful things in package management systems (such as the verify command detailed above.) Package management systems are there for a reason. Use them!
Simpli - Your source for San Jose dedicated servers and colocation!
when they trojan your MD5 checksummer?
Then your compromised system might apear to be clean. I have actually seen a system where that has happened. But the intruder forgot to trojan the rpm executable, "rpm -Va" revealed everything. But had the intruder trojaned the rpm executable too, that wouldn't have worked. The only secure way to use the verification tool is to boot from a readonly media and run the tool from there.
Do you care about the security of your wireless mouse?
Ok, lets see if I've been hacked... /dev/null
/dev/null with /private/var/servermgrd/servermgr_dirserv.lock from Mac OS X. What a bummer and its a brand new system too...
$ md5
d41d8cd98f00b204e9800998ecf8427e
So I put d41d8cd98f00b204e9800998ecf8427e in the search engine and it came up with 560 hits (compared with 3170 from google).
Now it appears that someone replaced my
Does the database have a way to flag files as being bad? Sa
When I put in 3ac9bc346d736b4a51d676faa2a08a57
I should get back:
*** Trojaned openssh-3.4p1.tar.gz ****
One thing that could make this useful would be a dns like interface...
host 3ac9bc346d736b4a51d676faa2a08a57.knowngoods.org || echo bad
Sun already provides this for Solaris.
s .p l
http://sunsolve.Sun.COM/pub-cgi/fileFingerprint
It contains information for:
Operating Systems
Solaris SPARC - 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.5.1, 2.6, Solaris 7 and Solaris 8
Solaris x86 - 2.1, 2.4, 2.5, 2.5.1, 2.6, Solaris 7 and Solaris 8
Solaris PPC - 2.5.1
Trusted Solaris SPARC - 2.5, 2.5.1 and 7
Trusted Solaris 7 x86
Most CDs bundled with Solaris 2.6 and later.
Patches
Nearly all released Solaris patches, including all SunSolve CDs to date. (4.0.11)
All Solaris 2.6/7 Maintenance updates.
All patches available from SunSolve.
Unbundled Products
Around 150 CDs with unbundled products are included. If you are missing any particular product, please feel free to send email and we will try to include it as soon as possible.
I like this utility. It's pretty handy, although probably not as effective as this database, unless you're running slackware, or another popular, but undatabased distro. :-)
If you could be told what you can see or read, then it follows that you could be told what to say or think - BoC
the poster mentions Tripwaire, but what about AIDE?
In additon to being a proper Open Source project, it allows for features that (last I heard at any rate) tripwire doesn't support, like a centralized checksum DB. That feature alone makes the tool superior (IMHO). For example it makes the verification process a lot nicer (intruder can't courrpt the local md5sum's because there aren't any).
You could start by subscribing to the forensics mailing list over at securityfocus.com. The honeypots list is also of interest.
Both lists have a fairly good signal-to-noise ratio, and there is a lot of good info to be had.
If nothing else, it's certainly a good place to ask that exact question.
You can sign up here.
Never trust an atom. They make up everything.
OK - debian seemed to have one version there - r5, whatever that is. How does it handle apt-get upgrades? If r5 is reffering to something like stable, then even stable changes over time (contrary to what some poeple think ;-). So do they take the checksums from a machine that was just apt-get upgraded last night, or what? If they mean an actual yearly or half yearly release, who on this world does not apt-get upgrade when there is a security fix released? So your system sure as hell aint going to match theirs.
/bin /sbin, /usr/sbin etc - do they have some alternative to HTTP for their database?
Then I can't imagine how you would be able to automate this, so it checks all the binaries in
Doesn't seem overly useful to me....
I found a fairly straightforward solution to this problem. I wrote a small wrapper around a known-good md5 function, compiled it and placed it in a nonstandard location. (Thus it doesn't have a widely recognizeable filesize or md5 to be detected and stomped) Then I wrote a simple shell script which checksums various critical files on a regular basis and tests the MD5 values against a record it keeps, again in a private location. Whenver a change happens, it sets off alarm bells all over the place, both in syslogs and on the console.
/etc/crontab, then checking each local shell script for a sensor and carefully overwriting my own nonstandard code - but if any attacker has that much free time on his hands, there's a limit to how much of a sensor I can implement.
On top of this I stuck in one small bit of shell script that allowed me to modify a file myself without setting off alarms - it simply recalculated the md5 value and updated the record files.
I suppose this is theoretically vulnerable to an attacker reading through
The nice thing about this code is that it also implicitly tests for corruption of critical files after fsck-triggering events like kernel panics or total power failures. (That's actually what prompted its initial writing) And it's remarkably trivial to implement, even more so if one simply copies an off-the-shelf md5 binary rather than compiling one's own wrapper.
"Doesn't seem overly useful to me...."
/bin/* /usr/bin/* for all packages for all distributions for all releases, or when do older things get purged?
Nor to me, for a different reason: what about those of us with CFLAGS= set to various strange funky optimizations in Gentoo? What about the Ports system in FreeBSD, similarly?
This thing does not have the potential to spread to all distributions or all unixen.
What about historical storage? Are they really proposing to store an md5sum for
Seems mad to me. Would be better off staying with AIDE instead, IMO.
~Tim
--
Rushing on down to the circle of the turn
We need file verification, too! Probably more so with some of the Windows/IE vulnerabilities.
Don't worry, you'll have that soon. It's called Palladium.
As my grandmother used to say: "Be careful about what you wish for, because your wishes might come true". Wise woman.
"There are already a million monkeys on a million typewriters, and Usenet is NOTHING like Shakespeare." - Blair Houghton
Will send you the files later. My address is 192.168.1.1.