RIAA Tracking Songs by MD5 Hashes

aSiTiC writes "Apparently RIAA has obtained some technical experts in their prosecution of file swappers. Currently they are tracking traded mp3 files from the Napster network by matching MD5 hashes. This seems quite interesting but I was under the assumption that identical hashes could be created with identical rips and id3v2 tagging. Now may be the time to update your illegal mp3 file MD5 hash sums."

Comments

MD5 Cannot stand up in court.

[Organized Konfusion]

The md5 hashing algorithm has been proven to contain flaws allowing two files to produce identical md5 sums.

Re:MD5 Cannot stand up in court.

[Urkki]

A bit of clarification is in order I think.

First of all it's very clear that two files can give same MD5 checksums. After all, MD5 is only 16 bytes (2^128 different possible). So if you have just 17 byte files (2^136 different possible), it's clear that on average every MD5 sum matches to 256 of all possible files.

It's just damn unlikely to get 2 files with same MD5, and if you wanted to brute force it, you would have to try average 2^64 different files before you found one with identical MD5 to another file. And this would take a long time (actually not that terribly long, a few years at most, and it parallelizes perfectly).

The page you link to implies that it's possible to "easily" fabricate a file that produces a given check sum, so instead of months of processing time, only days or hours would be needed to get a MD5 hash collision.

So all P2P users / software makers need to do to circumvent this, is to agree on a specific MD5 sum, then patch every file so that they produce this same MD5 sum :)

Of course the obivious solution for RIAA would be to use a more secure hash algorithm, with more bits. Unbroken algorithm with enough bits can't be faked, as it would take more than age of the universe to brute force it.

Though the basic problem with this RIAA method remains. If you rip with same software from identical CD digitally, and there are not bit errors at ay point, then you should end up with identical file, and therefore identical hash no matter how secure the algorithm is...

MD5 Hash

[fruey]

This seems quite interesting but I was under the assumption that identical hashes could be created with identical rips and id3v2 tagging.

The only way for two files to have the same MD5 hash is for them to both be encoded with the same encoder, from the same WAV file, with the same bitrate and all advanced options, and to have exactly the same ID3 information, the same filesize, and to be identical to the last bit.

Otherwise, the MD5 will be nothing like the same, for two perfectly identical songs where one has a spelling error in one field of the ID3 tag. I imagine for any one song, there are many many different MD5sums out there, although perhaps one or another good quality version would exists on hundreds of different PCs...

Re:MD5 Hash

[3terrabyte]

Many people will produce a file by ripping straight from a CD, which , given the same CD, will result in an identical source file.

No!! That's definately not true. Making a perfect rip is something you have to WORK at, which not many rippers do. Especially years ago. Check out ChrisMyDen's Uber Network on a detailed guide on how to make the 'perfect mp3'.

You need to use something like EAC's secure mode. It rips the cd twice and compares for exactness. Only then can you be assured your wav file has no errors.

Even if you can convince people to use the best mp3 encoding techniques (LAME 3.92 or LAME 3.90.2 -aps) I have still seen people refuse to use EAC, instead enjoying cdex, audiograbber, or (gasp) jukebox due to 'ease of use'. These ripper DO NOT make perfect rips, and will almost always make a different wav file each time due to the way it tries to make error corrections. Most people will not ditch their source either, even if there are errors. And everyone has a different scratch on their cd's.

Almost everyone encodes at 128kbps

This isn't true anymore either. Considering most of the lazy people out there download mp3's instead of make their mp3's, many of the rippers today do care about quality, and will rip in VBR or at 192. Release groups (where I would imagine most of the new stuff originates nowadays will rip at 192, 224, 256, or 320)

Md5 hashes are also used for....

[shione]

hmm Isn't that how k-sig, built into Kazaa Lite K++, works, by tracking MD5 hashes so ppl get exactly the file they want.

Changing MD5 hashes on songs to avoid RIAA would also lessen the effectiveness of K-SIG. Trading hashes of know working files was one of the ways ppl on P2p avoided downloading those fake RIAA files.

Re:What if...

[DrEldarion]

Yes, because for them to know that you have the MP3s, you have to be sharing them, which is the illegal part.

-- Dr. Eldarion --

Re:What happen if

[l1gunman]

Any modification, to ANY bit of the file covered by the hash, will change the MD5 hash (that's how hashes work). If you assume the hash includes the ID3 tag info, then simply editing the info (putting something in the notes field, for example) would change the hash.

On the other hand, if I were the RIAA attempting to identify common files in this way, I might be inclined to exclude the ID3 tag from the MD5 computation since it is so easily modified.

Any changes to the actual content, though, will ripple into the MD5 computation.

Short answer: "normalizing" the file for volume, or even chopping off a few seconds of trailing silence with something like CoolEdit will certainly change the hash and make it distinct from whatever their baseline hash value is.

Easy

[sprouty76]

Just take a random id3 field that you don't use for anything, and fill it with a random number. You can probably write a srcipt in a few seconds. Bingo, different md5.

The only problem is that a lot of file sharing software uses the fact that 2 files (from different sources) have the same hash in order to swarm the download from multiple sources. If everybody goes around intentionally making their mp3s have different hashes, swarming basically won't work anymore.

Re:gee?

[squiggleslash]

To put this in context, the RIAA was responding to the impression "Jane Doe" gave that the MP3s were rips of her own CDs:

The disclosures were included in court papers filed against a Brooklyn woman fighting efforts to identify her for allegedly sharing nearly 1,000 songs over the Internet. The recording industry disputed her defense that songs on her family's computer were from compact discs she had legally purchased.

Of course, the wording of the latter is ambiguous - it could mean nycfashiongirl meant she had downloaded MP3s of pieces of music that were also on CDs in her possession. A lot of amateur lawyers on Slashdot (ahem) claim this is fair use, and given it's non-commercial and wouldn't have an impact on the ability of the artist to make a sale, that may well be true.

(This wouldn't, though, be a defense for the central problem that she made all of these MP3s available for download by millions of anonymous strangers without the consent of the copyright holders. And assuming her identity is revealed and she is sued, if the "ambiguous" claim's alternative interpretation is correct, she'll be able to show the CDs to the Judge.)

MD5 sums and different encoders

[Psyborgue]

Pretty much no rip is identical.

First step: the *.wav is ripped. Using libcdparanoia, which i personally perfer, i find slight variation in size depending on the machine and cdrom drive i rip them on.
Second step: encoding on different machines, with different encoders, using different algorythms, using different levels of floating point precision, on different architectures etc... produces vastly different files.
Third step: sharing. Oftentimes an mp3 is downloaded 99.8% before the connection is broken. You keep the mp3 becuase mp3 is a sequential file format and you only lose a second or two of music. The rest of the file is intact.

Their md5 searching scheme could be circumvented quite easily by changing a comment in the id3 but they could get around that by cutting out the id3 part of the file when they make their md5sum.
The downside to this is that if you are searching for music on something like gnutella by the ***sum, the content would differ and you would not get as many results. Gnutella would not download from multiple sources becuase the file would not have the same signature.
Whatever the case, it is clear that some form of file obfuscation is now needed for safety online. Or we can wait for freenet to mature.

Similar story on BBC

[SuperChuck69]

http://news.bbc.co.uk/1/hi/entertainment/music/318 7695.stm

Re:MD5-hashes

[Gherald]

> This proof of RIAA is as good as the SCO evidences of greek language or bsd firewall code against linux

Uh, actually this is irrefutable proof. It will miss a lot of songs, but it is virtually guaranteed to not give false positives. This is much more solid proof than SCO had.

To think a month or two ago when SCO was insisting on an NDA many on /. were clamoring for some MD5 sums instead...

Obviously the RIAA's technical experts know what they are doing... its time to alter a few ID3 tags like the story suggested.

Re:What if...

[IpalindromeI]

you have to be sharing them, which is the illegal part

Actually that's not true. They only care about the sharing because it leads to what they really care about: people listening to music that they didn't pay for. If everyone who shared mp3s had bought every CD of the songs they downloaded, no one would care because they would have already paid to listen to those songs. The problem is that most people don't own all of the CDs for the songs they download, and the RIAA doesn't like it when you try to wriggle out of their money trap. If the actual sharing was the problem, the distribution itself, then we wouldn't have radio stations playing music either, because that also lets people listen to music they didn't pay for, but it's a bit different because you don't really get a choice of what you hear. But now if you go and start recording songs you hear on the radio, so you could listen to them whenever you wanted, you're getting into that grey area. Of course the RIAA doesn't really care about that because they know that radio quality is shit, so there won't be widespread radio recording anyway.

Re:What happen if

[1u3hr]

Short answer: "normalizing" the file for volume, or even chopping off a few seconds of trailing silence with something like CoolEdit will certainly change the hash

If that's all you want to do, much better not to use Cooledit, which has to expand and recompress the file to MP3. Use something like MP3Trim which can chop off any given number of MP3 frames, or normalise the volume, by operating on the MP3 directly. Much much faster, and no expand/recompress quality loss.

Re:MD5-hashes

[nolife]

I just did some consecutive rips of an audio track and compared the md5 checksums.

I did the same song three times. The first two times, all things were equal including all settings. The MD5 checksums were the same.

I swapped out my DVD/CD player for a different model. Reripped the track on the same computer with the same exact settings and the MD5 was different.

I am using Exact Audio Copy in secure mode and Lame for the encoding. The ID tags were recieved the first time and the same tags used for all three attempts (EAC remembers the disk).

I'm sure I could try many things like changing the read speed, comparing the wav files and not just the resulting mp3 etc.. but I do not have the time for more analysis.

Re:gee?

[gozar]

This is pretty common at least with iTunes. Most of the people will not change the default settings, so each cd rip will be identical, all using the same id3 tags.

Re:MD5-hashes

[henele]

If you read places like CDFreaks you'll see that extracting CD Audio is a mix of science and voodoo.

Theres issues of offset values (as with CD audio it is difficult to hit an *exact* location on the disk), plus the way the reader deals with C1 and C2 error correction, as well as how different extracting software interfaces with the hardware.

It would almost be safe to say two mp3s with the the same MD5 are one file copied twice (as opposed to two individually created mp3s), but that doesn't mean they are illegal...

Re:MD5 hash "posers"

[eric76]

Wonder if there is a utility for generating files with random content, but with the same hashes as another file?

Perhaps a reverse md5 hash generator which takes a hash and generates a file.

If that were possible, it would destroy the value of an MD5 hash immediately and everyone wouild quit using it faster than you could blink.

The purpose of CRC hashes is entirely different. They are designed to detect a burst of bit errors in a stream of data, the type of error that is most likely to occur in a network transmission. They are not meant for fingerprinting files.

I doubt that anyone with any degree of sophistication in cryptology would attempt to use CRC and MD5 hashes interchangeably.