Slashdot Mirror
Specialized MP3 Compression Hardware?
Syn Ack asks: "Does there exist any specialized hardware for the sole purpose of compressing audio into MP3 format? Without going into too much detail, I'm putting together a platform that has to rip and compress as many songs as possible in the shortest amount of time (ie compressing several thousand CDs). I was considering a large ~20 machine *BSD or Linux cluster of high end x86 or Alpha hardware solely for compressing digital audio into MP3 format. Before I go ahead and look at other solutions, I wanted to check the distributed minds of the Slashdot readers to see if they know of any specialized hardware for this purpose. Price isn't really a concern. Something like the dedicated encryption boxes/cards used on Web servers is the type of thing I'm thinking of. Anyone have any ideas?"
dunno about hardware, but why not put up 10-20 boxes with the above drive, or maybe sharing a tower to make cd-changing easy?
Malk-a-mite
I dunno if there's any hardware out there for that purpose, but if you can find an open source encoder and compile it with Intel's SSE or AMD's 3DNOW! (depending on machine type) you'd likely get a substantial boost.
Also, if price really is no object, for efficiency's sake, rip them all to wav first, then encode later.
If quality is a concern, use something like EAC (Exact Audio Copy - for Windows. There's a Linux/unix equivalent available, but I can't remember the name right now) when ripping. You'll get copies as accurate as possible before you start encoding.
Make sure you use the right encoder, and make DAMN sure you use the right settings!
LAME is considered the best encoder by most people, especially at higher bitrates. (I wouldn't bother with anything less than 128, but for my own collection, I do 256.)
If you are willing to do a little "roll your own" work, the best bet would be to use a DSP platform. I haven't seen any production platform for that, but the big three DSP companies (Motorola, Analog Devices, and Texas Instruments) all make evaluation cards which place the DSP on a PCI bus. Additionally, third party suppliers make cards, some with multiple DSP's. The latest Fraunhoffer encoder is claimed to run at 10x real time on a PIII 500, if that is fast enough. If not, use multiple DSP cards, and the server processor will have the job of submitting data for encoding, and keeping track of which cards have finished their encoding work and need more data. Fraunhoffer already have implementations for all three of the DSP's mentioned above, but didn't mention any hardware that was ready to go. Of course, by the time you do all that driver work, you could probably be finished with software encoding. --caudle
Imagine a beowulf cluster of.... oh yeah, we're talking about a beowulf cluster.. ;)
I haven't heard of anything that would be directly applicable to your situation.
I'm sure that there are DSPs that work well for encoding MP3s in real time, intended for MP3 walkmen and stuff.
But you are talking about really cranking them out at faster-than-real-time. Which probably means you want a bank of drives and a traditional PC cluster with whatever CPU will encode fastest. My feeling is that someting with 3DNow/KNI would work better, but I'm not sure.
Besides, general purpose computing is better. If you don't need all of the machines, you can put them to work for other tasks.
Gentoo Sucks
I doubt this is it - in this case, price probably would be a concern and 20 boxes would be overkill. I'm sure he's trying to build an encoding system for a commercial Internet radio service or other type of online music service.
The SoundBlaster Live! mp3+ was touted as having onboard mp3 encoding capability (through the firmware...it's the same card as the others). Specifically, it claimed a 5x acceleration up to 320kbps. As far as I can tell (I've got one), the acceleration is only used by the included mp3 encoding software.
The thing is, with that and a p3-500 I can turn an album's worth of wav's into mp3s in about, oh, 10 minutes (vs. about 50 minutes in software using BladeEnc).
I've got a friend with an 850MHz Athlon and a Kenwood TrueX 72X drive. With the Xing encoder he can do an album in 5 minutes flat...from the CD! The 72X drives are a MUST for mp3 encoding...the drive speed will almost certainly be your bottleneck.
Using my K6/266 and lame, I can encode MP3's in better-than-realtime. I'd suggest setting up a fileserver (NFS) with a few SCSI cd-roms (Plextors of course).. give it say 1 hdd per client encoder machine, write yourself a shoddy little perl something or other using cdparanoia to rip them... then setup 2 or 3 or whatever athlon's(or durons, a 1200MHz athlon should be able to encode a 6 minute mp3 in 1 minute or so, though i could be a little off on that) with a shoddy little perl something or other encoder script to encode them and delete the wav's when it's done all automagically.. so you just get to sit there swapping CD's. The tricky part would be figuring out how many cd-roms to use for ripping versus how many clients you have and how fast they are. so as everything stays busy all the time. I think something along those lines would be really easy to implement as well as cost effective. especially if you have some machines laying around already that could be used.
I don't know about the computer hardware, but one way to automate this whole thing might be to make a Lego Mindstorms-based CD Changer. Build up some sort of caddy that you can fill (with dozens, or hundreds of CDs), in some manner that the robot can grab them, and have the robot switch the CDs on you. That way you can keep the encoding running all night without having to resort to having someone watching dozens of machines 24/7.
Perhaps you can have a dedicated CD tower with 15 or so drives (SCSI, of course) and have the robot drop a new CD into whichever drive tray happens to be open, and press the drive closed.
Have the CD eject when done, the robot's touch sensor rigged to 'see' this and pop a new CD in. I think it might be worth the effort - you'll only have to keep the robot 'fed' to keep the encoding going!
Naturally this would help - if you want to do the encoding quickly you'll have to use your hardware 24/7. This is one way to keep it going. At the very least, it's a neat excuse to play with Lego!
There is quite a lot of specialized hardware out there, but you'll likely need to write your own drivers in most cases. Sigma does offer some kind of Linux compatibility - they have a workstation/reference design that claims to support Linux.
Sigma Designs
Darim
Optibase
Good luck.
Seriously.. Use commodity PC hardware. Even given specialized compression hardware, the price point you have to beat is so horribly low as to make it moot. I can encode 3 average CD's per hour on a dual PII-300, using LAME, at good bitrates.
1000 CD's would take two weeks, granted, but extrapolate the numbers. For $200 you can purchase a board/cpu combo 1.5 times as fast, and handle that load in just over nine days. By the time you spend $1000, you have enough processing power to handle all those CDs in 45 hours. What about ripping? Well, using the best CD ripper for Linux gives me about 1/4-1/2 the rated max speed of the drive. So, to feed each 'node' (This is batch processing, so the term is in quotes) at the rate it consumes wavs you would require at least one 40x CDROM drive. Another $80 should cover that. Need Ethernet and cases, so toss on some $25 Tulip cards and a cheapo $40 case.
We've spent $1,725.
Okay, to cover the final costs; Three 40G IDE drives should handle the compressed audio and the temp space required by the compression farm. Say $600 for those drives, another $500 for the server they reside in, and $200 in associated 100T network cost.
We're up to $3,025.
Whether to place the CDROM drives in the server or in the nodes is left as an exercise for you; Putting them in the server is more convenient, but you'd require $100 in SCSI hardware. Doubling the number of CDROM drives doing the rip will decrease the number of trips you have to make to change discs to every half hour, but at that point it is vastly less of a headache to go with a pair of 14 disc cabinets and change every 1.5 hours. Really pricey to do CD cabinets tho..
So, $3K for 500 CDs/day the cheap way, add $600 for some extra CDROM drives and SCSI hardware to make your life easier.
Oh. Everything assumes you've got someone changing discs 24/7. Tweak for the desired work day.
Someone else mentioned EAC for Windows; The equivalent for Linux/*BSD is called CD-Paranoia.
.sig: Now legally binding!
Screw the Plextor ... use the Kenwood 72x True CD-ROM. Under windows 98 I can get rip speeds of about 60x-65x. The CD-ROM has a buffer of 2MB and it blocks if it's empty -- I've ripped about 300 or so CDs with it and not one of the rips has skips in it (even my scratchy old Led Zeppelin CDS).
... I think the most specialized hardware you'll find for MP3 encoding (at this point in the technological timeline) is a hefty processor. Speed really counts, since it's just number crunching. I liken this to the good ole days of software compilation on 386-SX machines -- the faster your CPU the faster a program compiled. I'm sure you know, but the better quality encoding you do, the longer it takes to encode a given song. I usually use moderate/high VBR for all my songs (since I like post-processing using my equalizer) and have found that the only way to speed up the encoding is to lower the quality of the resulting MP3.
Using this CD-ROM, the bottleneck on my p3-800 system is the encoding itself. It takes only 4 or 5 seconds to rip a 20 minute song, but more than a minute to encode it. This was your original question, though
I've been using the AudioCatalyst product for about two years now, and it has been (and continues to be) the fastest and best sounding (don't bore me with comparisons of audio plots -- it's not the data that's in the MP3 that matters, it's how it sounds) MP3 ripper/encoder that I've found. CDex (http://www.surf.to/cdex) is a close second since it fits nicely on top of any encoder that you want. I've gotten comprable speed out of Cdex as from AudioCatalyst (http://www.xingtech.com)
He's not a DJ. If he was he wouldn't buy a cluster of machines to do something he only needs to do once. I bet he's trying to set up some kind of business that needs encoded CDs for such things as streaming. If you buy a 20 machine cluster, once all the major ripping is done you can convert half or 75% of them to streaming machines and leave the other half or 25% for future ripping purposes and general stuff.
anacron.
It looks like PCs are hard to beat. Here's a claim of a Fraunhofer encoder at 10X real time on a 500 MHz Pentium3. Maybe it's vapor, but if it's real it's certainly a lot faster than LAME 3.87 (beta) when I tried it on a 800 MHz machine (Lame ran at approx 2X using "-V 3"). I recall hearing claims of LAME doing real time on a 266 MHz PC... maybe it does with different settings.
PJRC: Electronic Projects, 8051 Microcontroller Tools
-
Physically loading the CD's into the readers. The cycle for retrieve CD, open CD, remove disk, insert disk into drive, retrieve CD, replace in box, refile box is slow & complex. I don't know of any system that automates this for jewel-box CD's; all autoloaders *I* know require the CD's be first loaded into special magazines which of course makes them pointless.
-
Reading the CD into the system. Even with fast drives we're talking several minutes of time to read in each CD. Bus bandwidth would also come into play - I'm betting that SCSI would be the way to go here.
-
Converting the CD audio to MP3s. Here you finally have a straight-forward read-compute-write process.
All three processes would be best tuned to each other. Given that the only means I'm aware of actually getting the CD's into readers is manually then one can assume a hard limit of loading & reading any CD is going to be about 5 minutes under best-case hardware conditions (fast SCSI reader etc.)Since a single load-person could likely service two readers alternately this would make a work station. This work station would then produce 2 CD-datasets every 5 minutes. Assuming about six hours of loading per day one would achieve around 144 CD's of data loaded per station per day.
Assuming two CPU's per station (1 CD-ROM/1 CPU) one would have 20 minutes to process each CD over 24 hours in order to keep pace. As I believe this rate is achieviable with a current fast system then you have a well tuned solution.
Workstations consisting of 1 load person with two fast PC's = ~140 CD's per day processed
No need for custom hardware, no need for clustering, all off-the-shelf stuff.
I don't read ACs: If a post isn't worth so much as a nom de plume to its author then I wont bother either.
This review http://www.tech-report.com/reviews/2000q3/kenwood7 2x/ found significant problems with the quality of the Kenwood's Direct Audio Extraction. They were particularly a problem for a less than new disk which may not be an issue for the person asking the question.
May not be very helpful, but you can save alot of money and time (no need for clustering) by just using DistMP3. Its a distributed client server system that offload encoding to many computers. Like distributed.net.
Unless your use is tied to current MP3 technology ONLY, you're being very short-sighted by making MP3 compression your primary concern.
Your concern should be about how you can read and store as many CDs in raw format so you're always ready to convert the library to the current audio format, be it MP3, Ogg Vorbis, WMF or whatever else tomorrow's standard may become.
couple the cddafs of BeOS with the gogo encoder and an SMP box, and you've got a great encoder.
Many manufacturers make 100, 300, 500 CD jukeboxes for use with audio CDs. Is anyone making one with a data-out option or an ethernet port for remote control, etc? How about using the TOS-link fiber out to send data to the PC for reinterpretation?