Slashdot Mirror
Best Practices for a Lossless Music Archive?
Sparagmei asks: "I'm a big music fan, and I like listening to the music I own on various pieces of digital gear. Right now, my library's at about 20,000 tracks, ripped from CDs to MP3 at 256kbps (enough that I can't tell the difference on my low-end playback gear). However, with the MP3 judgement rippling through the world, I'm interested in perhaps moving to a different compression standard. Before I do that, I'd like to ask a question: what lossless format would you recommend for making a digital 'master library' that could be (relatively) easily down-sampled to a compressed format?"
Important factors would be true losslessness, filesize (smaller than PCM WAV would be nice), embedded metadata (ID3v2-like), existence of automated ripper software, and (to a lesser extent) an open-source implementation of such software. Widespread playback implementation of the lossless codec is not an issue for me; the lossless library would likely be burned to archival DVD media and stored after being down-sampling with the chosen compressor. The reason I ask is this: I've got a 20,000-track re-ripping job ahead of me. I'd like to do that just once, lossless, so that years from now, when I decide to jump from Vorbis to 'komprezzor_2039_1337' or whatever, I don't need to drag out the old plastic discs. Thanks!"
The Free Lossless Audio Codec sounds perfect for you.
:-) Both hardware and software support.
(all answers below are quoted from wikipedia's FLAC page).
Important factors would be true losslessness,
A digital recording (such as a CD) encoded to FLAC maintains the quality of the audio perfectly.
filesize (smaller than PCM WAV would be nice),
Audio sources encoded to FLAC are typically reduced in size 40 to 50 percent.
embedded metadata (ID3v2-like),
with support for tagging, cover art and fast seeking.
existence of automated ripper software,
Yup, lots.
and (to a lesser extent) an open-source implementation of such software.
See above.
Widespread playback implementation of the lossless codec is not an issue for me
Well, bad luck, you're going to get it anyway
Also, consider SHN, (although it seems superceded). I'll also mentiuon wave pack - because it uses an interesting approach (splitting the file into a small lossy standalone & a lost bits diff). don't bother with Apple's lossless format - it's going nowhere.
There are shills on slashdot. Apparently, I'm one of them.
I made the mistake of ripping my small collection to MP3 (some to MP3pro...please stop laughing) the first time. I made it about 1/2 way though the 300 or so CDs I have before I realized that I wan't happy with the format (could hear artifacts) and knew I wanted a lossless that I could transcode to the format du jour.
I went with FLAC, and ripped 'em all. I'm using media monkey as a filing system, and am transcoding as necessary for portable apps. I'm without media server at the moment, so I can't help with streaming and such, though I'm going to be interested to see what others are doing.
Is it just my observation, or are there way too many stupid people in the world?
Bearing in mind that you're not going to save that much file size using lossless storage, and that you already have an "archive" of CDs in a box in your basement (or wherever), is it really worth the hassle of creating another lossless copy that'll take up even more space?
If you're planning on re-converting from these lossless copies, it sounds like you're going to be doing a *lot* of work based on some second-guessing of where you'll be in 5 years time; and things may have changed then.
"Slashdot - News and Chat Sites Deviant". (Click "homepage" link above for details).
Since FLAC was already mentioned, I'll just suggest you try Monkey's Audio. It's lossless, usually compresses better than FLAC, the source is available (not sure what license though), supports tags, and basically does everything you want. It's probably not as widespread as FLAC, but that shouldn't be a problem in your case.
Using a torrent client which has RSS support.
Next question?
FLAC doesn't compress the absolute best of the alternatives, but it's 'good enough' and is widely supported, even directly on some portable devices. You won't wake up one day to find out that FLAC support has all but disappeared because the original developer lost interest (since the source is out there, unlike many alternatives). You will also be able to trivially transcode FLAC to Vorbis with meta-data intact, and do it FAST. (not a unique property, but well supported with FLAC/oggenc2).
Belief is the currency of delusion.
I suggest you back on a Hard disk; It's about the same price per Gigabyte, smaller, and i guess easier to maintain; Last but not least : it's more durable.
Yes, I'm left. You have a problem with that?
Don't use FLAK or some other monkey sound formats. Go with a real standard, like Windows Lossless, or
Apple Lossless, they just sound sooo much better. But you do need high end Bose equipment and gold
plated Monster cables to really bring out the warmer and fuller mid-range and the increased bass response.
Forgot to mention that you'll probably want to go to hydrogenaudio instead of /. for these sorts of questions.
Belief is the currency of delusion.
I had a similar problem - a huge library of CD's ripped to lossy mp3, which I decided to re-encode. I used iTunes and Apple lossless (ALAC), for iTune's excellent interface, search and catalog facilities and to access CDDB data. I can play the files on my iPod and the codec is widely supported (by either Quicktime or libavcodec on Linux) on the desktop through iTunes and Mplayer or VLC. For archival, simply copy the iTunes database file (or the xml copy) and the individual files to DVD to preserve the files and other data. Most files are in the region of 50% of size of an uncompressed file which is comparable to other lossless codecs (and impossible to exceed for truly random data).
Oh, and see the comparison.
Sorry to jump on somebody elses post... but can anyone speak to the quality of hardware based devices that can rip CDs straight to FLAC format such as the products offered by olive.us?
Most hardware devices I've seen either a) can't rip, b) can't convert to FLAC, c) can't burn CDs from FLAC, or d) are just wireless devices to send the audio back to your computer's crappy speakers.
Anyone know of any other great sounding devices that rip, convert, and burn?
sig.
Parent was being sarcastic, was it lost on you?
He couldn't see the sarcasm because his monitor isn't using monster cables.
Don't lose it. No matter what Eminem may tell you.
The last word in lossless archiving...
My suggestion is to rip each CD to a bin/cue type format (with toc) and then burn your archives onto some fairly cheap optical media. I suggest CD-R. Then, if you ever need to restore the files, just insert the discs and use some ripping software to extract the data again. As an added bonus, your archival copies will play in a wide range of audio equipment. Glad I could be of assistance.
Yes, I agree with the parent. Use those most excellent formats for storing your music or else use PKZIP for MS-DOS and then copy the ZIP file to a IOMEGA ZIP drive for safekeeping
Everybody sit down. Did you see the line about Monster cables and Bose?
Parent was trying to be funny. He wasn't, but don't go off the deep end.
Overrated is more appropriate, though Troll probably will be the choice of the Mac fanbois for the Apple Lossless dig he included.
Is it just my observation, or are there way too many stupid people in the world?
I've been ripping all my CDs using cdparanoia and encoding with FLAC. I keep two copies of all the FLAC'd suff on separate hard disks, and verey year or two, when I buy new hard disks, I migrate the data to the new disks in case the old ones fail.
There's no point in burning to DVD for "archival" since DVD is too unreliable. Anecdotally, DVDs seem to last only a few months to a year or two. Perhaps good quality tape archival would be good if you need the security? But really, hard disks are so cheap nowadays, it's feasible to have many PeeCees with new, high-qulaity disks in your house and to have multiple copies of the data.
Through the magic of bash scripting, I can produce ogg/vorbis and mp3s of all my music whenever I like. My old Athlon XP 2000+ used to do this really quickly. I haven't tried it on my new box yet.
Stick Men
In my opinion, there is no other choice for lossless audio compression than FLAC. Since it was BSD licensed a couple years ago, it has been rapidly taking over the world.
It appears to have overtaken all other lossless formats in popularity online, in a short time.
It is the only one seeing significant adoption in multiple brand of hardware (MP3) players.
I'd say it has a brighter future than Vorbis, even though it is at a several year disadvantage.
I've found that just about every audio program I use, has silently added FLAC decoding support while nobody was looking.
With all of it's positive attributes, and support, FLAC is well on its way to becoming ubiquitous, and actually replacing WAV/AU/RIFF/AIFF/etc. files as the standard lossless storage and interchange audio format.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
I use flac because my Squeezebox supports it. So does XMMS. So does Winamp. Nero and Roxio support burning audio CDs from it (with appropriate version/plugin).
I'd been down the mp3/Ogg Vorbos road but found myself transcoding from one lossy format to another which had to stop. With flac I have the lossless copy to transcode from. I rip on Windows using dBPowerAmp and with AccurateRip I feel that when it says it's accurate, it is. I've seen people start to rate dBPowerAmp as good or better than Exact Audio Copy.
My wife has an iPod and I finally have her ripping to Apple Lossless. I figure Apple Lossless is better than lossy AAC, even if it is proprietary.
Check out Flac2MP3 which I'm using to generate a metadata-complete MP3 image of my FLAC files. Whenever I add a couple of new albums to my archives, I run a shell script and the MP3's are available for my iPod. Album covers from Amarok don't transfer over, but it's a plenty good system for me.
Use of the term "best practices" indicates that you've already assumed that whatever you're doing is wrong, and that whatever someone else tells you to do is right.
"Best practices" is to IT what the "zero tolerance" concept is to schools - no questioning, no thought required, simply doing whatever the current meme dictates.
I want to delete my account but Slashdot doesn't allow it.
Seriously. What's the point? What are you trying to accomplish? You can't hear the difference at 256kps lossy versus lossless, so why waste you're life converting your already lossess music archive from CD form to harddrive form? It's not like you're going to be transcoding all that often, if at all. I encoded my entire CD collect 7 years ago as 320kbps constant MP3. I'm thinking about re-encoding it, because there's no point in having the files that big. 256kbps variable mp3 is would probably still be more than enough. And if you think that someday you can transcode to something higher than 320kbps, I doubt you'll be able to tell the difference if that ever happens, because your hearing naturally gets worse over time.
Futhermore, if you think Alcatel-Lucent v Microsoft is going to change anything, you're delusional. MP3 is going to stay. Just like how LZW patent did nothing to GIF. No one is going to abandon MP3, because the public isn't going to buy a device that can't play their MP3 collection. Nothing will change, and FLAC and Ogg will remain forever an asterisk.
Pioneer PD-F1009
Or something similar should do the trick. I recall seeing a 500 disk model from one of the home audio manufacturers once.
It will save you the job of ripping all those tracks and keep your material safe, clean and organised ready for play whenever you like.
Is there any linux software that will convert flacs to mp3s and preserve the tags?
I do RAID on Linux all the time, and I've never had problems with kernel versions causing incompatibility, or system flakiness. What kind of RAID are you using?
GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
Dunno what judgement you're on about mate, but MP3 is free for personal use.
I'd use FLAC - 20,000 tracks isn't really all that much, and storage is only going to get cheaper. Pulling numbers out of my arse gives me 300 meg or so per CD using FLAC, so assuming you've got 10 tracks per CD that's 600 gigs. That'll fit on a couple of hard disks for £150 or so (UKP), although you might want to back up your stuff onto DVD also. Or you might want to find a backup medium which has some chance of being readable in a few years. Assuming you actually bought the CDs the data came from you might be better off hanging onto them and just having the FLAC files on your hard drive. You can play them as-is using VLC or probably get plug ins for Winamp if you like bloat and non-standard interfaces.
You should really look at http://www.hydrogeaudio.org/ to answer tons of questions. I personally am using EAC with a React add-in that allows me to rip to FLAC and AAC at the same time. Tags and organizes in folders all at the same time. The nice thing about EAC (or Exact Audio Copy) is that you know you are getting an excellent rip without listening through your hundreds or thousands of tracks for quality control.
ive always wondered why people dont just use drive/folder compression and keep them as raw wav files? no special codecs needed. no converting. its just there in its original form, and im sure the compressed size must be similar? anyone do this?
I'm using flac for our entire library. However, I'm at the point of seriously considering a switch to ALAC. I use some custom scripts for ripping the tracks on OS X, tagging them with audiotag (http://www.tempestgames.com/ryan/), and then some mirroring scripts that check filestamps and output a mirror of a selected portion of the flac tree in a lossy format.
I'm possibly switching to ALAC because iTunes will play it back natively, and I can use it under FrontRow in a Mac OS X media center frontend. The only reason I haven't switched is because I'm currently combining aacgain and the open source alac decoder to create alacgain. Once I've tested that iTunes actually obeys gain tags on ALAC, I'll move ahead with development.
I will go with the flow and say Flac is the most used open source format for lossless. However I will digress from the flow and say most MP3 players don't support Flac. Since the widely used iPod is really here to stay, I have to say Apple Lossless is a good solution. No the compression is not as good, but it's more convenient, like it or not. And the compression not being as good doesn't mean there's a difference in output quality, it's lossless or it's not? :) So the super file format for losslessness should be on what is convenient to you, not what takes more or less room. If your players support lossless in Flac, then go ahead. Just pick what you want.
After having ripped most of my collection in Lossless, I however can safely say one grow tired of it. Some materials, you want in lossless, but I do not think it's practical to have everything in lossless. I got a lot of ripped CDs and a lot of ripped vinyls. I start by doing them in lossless, I archive them 10-15 CDs in one DVD in lossless (no not in the best Flac compression, something that compresses quickly please, as long as it's smaller than raw), then I switch the heck out to some other format (Mp4 vbr 128+) for me, Ogg if you want to go other ways, again what suits your particular players)
To rip CDs, please no normalization. Please. No volume change whatsoever, no "post-processing" of any sorts. Once you convert from lossless to your other format, you can then decide to apply some or not, at your leisure. But I see little point in using lossless format to change the volume of your tracks. And please use some software that correctly checks your rip. Exact Audio Copy for Windows is a good choice, and say you are not afraid of waiting. If there were errors, clean up your CD then try again. If your CD doesn't support basic functionality to properly rip with your software, consider buying a new CD player that supports everything. The worst thing I couldn't hate more are clicks and ticks (or worse, garbage fits).
For Vinyls, I clean up the vinyl as best as I can with demineralized water and cotton cloth, then I record it at its proper speed (or else you will not get the same frequency response) at -6db peak volume, make it go through a de-popper (very lightly, more you remove, worst it sounds), make it go through a de-noise algorithm (using the ending groove for my 500ms sample of noise, as it's usually quite clean, again very lightly, like a -3db reduction, or else it makes everything sounds electronic), I remove the DC (there's one with your equipment unless it's gigantically pro), remove 50Hz frequencies at -6db (depending on the frequency response of your cartridge and equipment, my Grado limits at 50Hz) and maximize the volume with a different volume for every side of the vinyl (as every side is engraved differently at usually different volumes), just enough to get some limiting on maximum a peak per 2 minutes.
I guess that basically covers my processes and what I would do (and don't do) for a collection.
The thing about dbPowerAmp is that the free version does not do the "secure ripping" for which the software is so revered. The real question is, how well does the FREE version of dbPowerAmp rip compared to EAC?
CD is already lossy. It truncates the audio to just 44khz 8bit stereo.
I'm not sure if this is the best thing to do, but I decided to just make disk images of my CD's, and store them on an external HDD. Then I use iTunes to rip them to 192Kbps AAC format for my iPods. I figured out that I could fit about 400 CDs, more or less, along with the ripped files, on a 250GB drive.
The disk image files themselves are made with Roxio Toast in Digidesign's Sound Designer II format. Yes, it's proprietary software and a proprietary format, but the likelihood that this format will be readable for a long time to come is high. And if it isn't, I can always remount the images and convert to another format.
I'm pretty sure that when I originally started doing this, I tried it with Apple Disk Utility, and had unsatisfactory results (though of course, I can't remember why).
I think it's important to get an image of the CDs as they actually shipped. If someone an suggest a more open method of doing the same thing, I'd appreciate it. The only reason I'm doing it this way is because it was easy for me to get started. I got to about 150 or so CDs, starting with my newest albums, before I realized that most of the rest of the older stuff I didn't care so much about, anyway. My CD collection isn't very large as it is, so I'm not especially worried.
I don't ditch the CDs. I do this so they can be stored and I don't have to handle them so they won't get damaged. Now if only I could do this as easily with DVDs!
FLAC should be your first choice but if you go mp3 I prefer LAME using VBR @ 'q 0'. Average bitrate on my library is between 240 and 280Kbps. Playing through my Squeezeboxes on any system from low to high I can NOT tell a difference. I have even tested a variety of interconnects including making my own. Of course you still need to downsample for most portable. Decoding high bitrate VBR files will suck your battery dry.
44KHz, 16 bits per channel, not 8 bits. And the term "lossy" would depend on the input data; if your input source is 44KHz, 16 bits per channel, 2 channels, CD will store it with no loss.
FLAC is great and all, but you only save about 30%. That's a significant help, but it's not a killer feature in my opinion. Uncompressed WAV format (RIFF storing PCM) has one significant advantage: it is such a trivially simple file format that any junior programmer can write a parser for it in an afternoon. fread(), a pointer cast, a linked list traversal and there's the bits. You don't need any libraries other than standard file I/O. That makes future support and migration a zero-risk proposition.
A nice feature of audio is that our requirements are scaling much slower than our tech. It takes about 100K to store one second of one channel of 16 bit, 48 Khz uncompressed audio. People will always argue about how many channels they need or if 96Khz is actually any better, but we aren't going to see real demand for 10X storage/bandwidth requirements for audio in the foreseeable future. Meanwhile storage and bandwidth are going to continue to get cheaper by factors of 10.
I expect FLAC to go the way of Stacker. It's a great tech that helps bridge the gap between the time when it was just too expensive to store so much data and the near future when it will be so cheap that it doesn't make sense to bother compressing it. I loved Stacker back when I had my first (5 gig) hard drive. Now that I'm pushing a terabyte, it would be silly to put up with the risks and the hassle of drive compression.
Disk space is ridiculously inexpensive these days -- a good 500 GB hard drive will set you back under USD 140 when on sale. You didn't specify how large your collection is in terms of uncompressed space, but let's say 20,000 tracks represents about 1,300 albums on CD (15 tracks per CD). That's about 800 GB (600 MB per CD). Four 500 GB drives (two for data, two for backup ... you *were* going to have an on-line backup, right?) and you're set.
As other replies on this thread have observed, lossless compression recovers about 30%. Is your time and worry converting to a lossless format worthwhile? Tagging, yes, but compression for your reference collection? To me, the answer isn't clear, and as disk prices continue to plummet, the answer will sometime soon become, "don't bother compressing."
Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
flac all the way buddy. it more than meets your criteria. and its open source. can't get any better imho.
http://www.ronpaul2008.com/ Ron Paul for President 2008 http://www.infowars.com/
It depends if it's worth it or not. It also depends on your equipment.
Look at the specifications of your cartridge, if it's 20Hz-50KHz for example, double that to be conservative. Remember analog is discrete, 50KHz means you can get a pure 49.999KHz sound. Digital is quantized, 50KHz would mean you can get a pure 50KHz, then a pure 25KHz, pure 16.67KHz, and so on. To get a proper recording, and get proper quantization, 2x or 4x the maximum is adequate, I would go to 96KHz. Or if you have the equipment, try 192KHz.
End result would be 16 bits probably, so if you record everything at -6db and have equipment that do true 24 bits (like Motu Traveler, again, look at specs and graphs to know if your equipment is true to what it advertises in your recording software), once you will normalize or limit the volume to full range, it will give 20-22 bits of true bits, hence reducing at 16 bits will only give you quality, not approximations. But that's later.
Then, post-process. Add your filters (again: less is better, no filters if not required, or else it sounds electronic and that's the worst thing you want to do). And finally, convert in the end-result (44KHz/16bits for CD, 48KHz/24bits for computer-based, 96KHz/24bits for DVD, 192KHz/24bits for DVD-audio if you got it), using the best dithering (not the strongest, the best) and even slight noise shaping, algorithm that's almost done for vinyls.
As far as records go, 80's and early 90's digital is the worse. Some 50KHz (no not 44 or 48, I mean the original 50KHz equipment) were decent, but most had no dithering, and very bad end-result quality, a lot of high frequency ringing and harshness. I would take less quality to these. 44KHz would be enough for these, no matter. Analogic records, I'd go to 96KHz for personal use, and for true archiving I'd go at 192KHz 32bits float, but that's for pro job and for that, there'd be a tape instead of record. Pre-50's, I'd revert to 96KHz. 78RPMs, I'd revert to 44KHz.
I assume the whole point of this is to be able to create _exact_ copies of your CD's? Well, this isn't possible on linux AFAIK.
To see why, try this:
1. Rip a CD to flac
2. Create a CD from the flac files
3. Rip the burnt CD again
Note there are differences between the 2 ripped files. If the process was truly *lossless*, the 2 rips should be the same. Yes?
I believe it is only possible to have the same rip twice under windows using EAC.
I do the exact thing you're describing. I rip to WAV with Exact Audio Copy, compress to FLAC (with highest compression level), convert the FLACs to OGG to keep on hard drive for listening, and burn the FLACs to archival DVDs. I use Taiyo Yuden DVD+R and burn them at half their rated speed (plus some other anal-retentive habits I've picked up through trial and error), then test their quality with Nero CD-DVD Speed. I burn the DVDs with sequential labels (music-01, music-02, etc.), and after burning a DVD, I keep a catalog of its contents. How? "dir /s >music-01.txt". Couldn't be easier.
To me it seems like the great features of your setup is your softare, of course made possible by FLAC. Now if you wrote some software that handels this, as you sort of implied, hey great you ROCK!!! Can you share it? or does it belong to your employeer if you did it at work?? Or is it software anyone can get open source or closed? Of course I am talking about the part where you use unique ID's to "create a Artist/Album/Track hierachy which again can be changed at at time fairly trivially" Trivially being the key word.
Those who can, do.
Right now I standardized on Monkey's Audio, I love how it works. But if the next version of FLAC outperforms APE in terms of size, required power and ease of use, I will just fire up my dbPower Music Converter and batch-convert the APEs. I could have done the same from any codec to any codec. This is why the question sounds to me like "Should I put my computer on my desk or under it?" to which the answer is: Pick a place, and if you don't like it, move it! Don't bother to Ask Slashdot about it!
I've been wanting to do this for a while too. I'm in the same situation with about 10 gigs of MP3s (r3mix or preset extreme) I have about 2,000 CDs I want to (re-)encode, and only want to do it once.
:)
My ideal setup would be:
Have a bunch of lossless files on a portable harddrive
Have a portable media player that can play high-quality lossy files
Have an "update" process, that when I move music to the portable media player, automatically encodes it to the lossy system, so I can fit more onto it, but keep the lossless files on the harddrive for reference.
now, iTunes seemed to do that for me, with Apple Lossless/AAC, and from what I've read (I've not tried it out), it seemed to have a transcoding setting as above too. What I'd like to know is, confirmation of whether it does or not, and whether any other solutions have the ease-of-use and convenience of doing the lot automagically when I want. When I want to sync my music, I want to drag a bunch of files and have it know i want it to change it from lossless to lossy, and not have to fiddle with shell scripts or anything else like that. I'm all for low-maintenance solutions.
And hi to any PJB-100 users out there, I'm finally outgrowing mine and going for a player smaller than a brick
Thanks for that, very informative. I've got a friend who wants to back up a rather large vinyl collection and he'll find this most helpful.
The q switch hasn't worked since at least version 3.96.1 of L.A.M.E. An explanation I heard from one of the developers is that it used to control the termination of some loop, but that loop is no longer used, to paraphrase from my marijuana riddled memory =D The -V X switch will turn on the appropriate things with no need for other switches, other than the recommened(by consensus on hydrogenaudio.org) --vbr-new switch, which invokes the new VBR algorithm and is alleged to be of higher quality. Now before you freak out and re-encode your mp3 collection(a benefit of using FLAC), its not like its some earth shattering improvement. I recommend FLAC + EAC + foobar2k if you are using windows.
Flac is great because of the license, freely available tools and because my digital audio players ( 4 SqueezeBox network players at home and an iRiver H140 with rockbox on the road) support it natively. Tag support is included.
- 14697.html
The great thing about any lossless format (as long as you have the uncompression tool) is that you may at any time choose to convert to any other format without loss. I regularly convert Monkey audio downloads to Flac without loss.
That's the real problem with MP3 library's you can never move to a different format without losing extra quality (unless you move to a lossless format which is - of course - totally pointless).
Many people maintain a separate mp3 tree, which they can use on any portable players where sound quality is not that important. Tools exist to automate this:
http://forums.slimdevices.com/archive/index.php/t
X.