Ask Slashdot: Best Kit For a Home Media Server?

First time accepted submitter parkejr writes "I started off building a media library a few years ago with an old PC running Ubuntu. Folders for photos, ogg vorbis music from my CD collection, and x264 encoded mkv movies. I have a high spec machine for encoding, but over the years I've moved the server to a bigger case, with 8 TB of disk capacity, and reverted back to Debian, but still running with the same AMD Sempron processor and 2GB RAM. It's working well, it's also the family mail server, and the kids are starting to use it for network storage, and it runs both link and twonkyserver, but my disks are almost full, and there are no more internal slots. The obvious option to me is to add in a couple of SATA PCI cards, to give me 4 more drives, and buy an externally powered enclosure, but that doesn't feel very elegant. I'm a bit of an amateur, so I'd like some advice. Should I start looking at a rack system? Something that can accommodate, say, 10 3.5" drives (I'm thinking long term, and some redundancy)? Also, what about location — I could run some cat6 to the garage and move it out of the house, in case noise is an issue. Finally, what about file format, file system, and OS/software? I'm currently running with ext3 and Debian Squeeze. Happy with my audio encoding choice, but not sure about x264 and mkv. I'd also consider different media server software, too. Any comments appreciated."

Comments

No reason to change from H.264

[InterestingFella]

Why would you change away from x264 and mkv. They are the industry standards. Not just in computers, but every way in the distribution chain. Going about it for some FOSS reason is just stupid because they're only for your own use, not for distribution. You would be either spending double the space or get half the quality by going with something other than H.264, and on top of that you introduce yourself additional problems because they are not what everyone uses.

Re:No reason to change from H.264

[Ltap]

Actually, Matroska has a number of good advantages:
1: More open than MP4 -- it has none of the ugly MPEG-LA overtones.
2: More codec support than MP4.
3: More consistency -- there aren't PS3-oriented versions or AppleTV versions of Matroska. MP4's device support might be wide, but when every player and device seems to have its own version with its own ridiculous, restrictive standards, it doesn't really qualify as supported at all.
4: Content management: Matroska is the easiest to mux and makes it great to work with alternate audio and video streams, subtitles, etc. You get the widest range of options of any container out there.

While Matroska might be used by some because it is perceived as more "elite", there are many very valid reasons for using it, mainly revolving around flexibility and openness. As someone who has worked with both containers, I can say that Matroska far more easily delivers what I want and the tools for working with it are generally free (in both senses), more usable, and more powerful.

Re:No reason to change from H.264

[Telvin_3d]

1) I've never had problems with using Handbreak for chapters. But one anecdote is about as useless as another for this kind of thing.
2) Only system I've ever found that supports .mp4 but not the subtitles is the PS3, and all indications are that it was deliberate. So fuck Sony and the horse they rode in on (slung underneath). On the other hand, PS3 doesn't support mkv at all

3) This is not a good thing if your goal is to play back the content on any system but the one it was made on. Ever gone internet hunting for that one weird codec that you used for a few months a couple years ago? No? Me neither because I'm not dumb enough to think that 'can jam anything into it' is a good thing in a media format.

4) This is... debatable. Both formats are open standards and open source. You can look at the specs and the code for either. The patents for .mp4 are known and need to be licensed if you are a large commercial operation. The patents for mkv are god-knows-what and may or may not get eaten alive the first time that the patents ever become important. No one knows. Pick your poison.

Re:No reason to change from H.264

[Ltap]

I find it amazing that you are even trying to argue that "fewer supported codecs" is a good thing. MKV isn't MKV isn't MKV -- it's all about the codecs you use, and that's what it should be about. The container is a way of gathering together media streams, but doing something like designing a player and assuming that, for a given container, there will be only a single or a couple of codecs (when that container actually supports many) is ridiculous. If your player can't examine a container it theoretically supports and find out which codecs it uses, then you shouldn't use that player because its manufacturers made far too many underlying assumptions. If you are arguing that MKV allowing unusual or poor codec choices means people will make them, then yes -- some people will, for various reasons. But does that mean people shouldn't be allowed to make a decision just because you don't think it's a good idea, based on your usage case? If there's anything FOSS has taught me, it's that many people will have truly unique circumstances which require very specific things to happen to reach a proper solution and flexibility is everything. What you seem to be arguing is that no one should be allowed to use a hypothetical ideal container in ways you don't want because you consider those improper.

Device support might be important to you, but I've just never understood why people contort themselves into making decision which are, frankly, idiotic simply because they have a crappy DVD player or something which only supports a few things in a handful of configurations. The most sensible and flexible solution is ... tada! A PC hooked up to a television with a cable, where the television is just yet another display device, no different from a monitor. Trying to wrestle with a DVD or (ugh) Blu-Ray player is just asking for some manufacturer to screw you over.

What I care about most is media support. What can you fit in? MKV's subtitle handling is simply the best around bar none. You get complete flexibility to store hardcoded or softcoded subtitles in any way you want and you can easily do the same with audio, something which cannot be said for most containers. Unless you are trying to square the circle by wrestling with uncooperative devices you are unwilling to give up, there's really no reason to use anything else, at least in the present.

Re:No reason to change from H.264

[Osty]

Others have touched on your other points, so I wanted to address this:

Switch to a standard .mp4 container. Much better supported on hardware or software. Some day you will want to be able to stream from your server to a thin set-top box or load a file on your kid's phone. On that day .mkv will make you cry.

Plenty of thin set-top box clients play mkvs already. Devices from Western Digital (WD TV Live), Netgear, Seagate, Roku, Popcorn Hour, Boxee, and many others all support mkv out of the box, with header compression support, subtitles, chapters, multiple audio and video streams, and some even support 3D (not the new mk3d format yet, but SBS works) and will play subtitles correctly. Most mkv files contain MPEG2, h264, or VC-1 video and AC3 or DTS (or the newer Dolby TrueHD, DTS-HD, etc) and all of these players handle those just fine. Don't blame the container for being flexible enough to allow any random codec. Blame whatever crap source you stole your videos from for using a random codec. Those of us archiving our DVDs and Blu-Rays will either encode in H264 or remux the original MPEG2/H264/VC-1 streams untouched and have no problems.

If you want to move videos to a phone, that's easy enough to do. The beauty of being a completely open container format means that it's trivial to demux Matroska containers into their component streams, which than then be remuxed into mp4 for devices that suck. Since you'll probably want to down-res the videos anyway for handheld formats (even on tablets you won't want higher than 720p), there's no reason to keep the originals in mp4. Keep your original, untouched videos in mkv and re-encode at lower resolution and bitrates into mp4 using Handrake for mobile devices.

Divx (yeah, whatever, they're still relevant) has adopted MKV as their HD container format, and the proliferation of "networked media tank" devices plus Matroska's openness makes it not only relevant but desirable for long-term video storage. Using 5+ year old devices like Xbox 360 and PS3 as your benchmark for what containers to use would be a bad idea (Xbox still doesn't even support 6-channel AAC in mp4, never mind supporting AC3).

Arduino.

[Anonymous Coward]

Just kidding.

Larger disks

[Anonymous Coward]

Larger disks. 4 TB should be available very soon (maybe now?).
http://www.extremetech.com/computing/108665-hitachi-ships-worlds-first-4tb-hard-drive-sticks-it-to-thor

HP Microserver

[quenda]

For a slightly more sane solution than rackmounting at home, consider the HP microserver.
Very low power (12W CPU), small, quiet, cheap, server grade, no Windows tax, holds four pluggable 3.5" drives plus optical (which some people swap for a 5th HDD for RAID5.)

http://blog.thestateofme.com/2011/05/14/review-hp-microserver/
http://www.silentpcreview.com/HP_Proliant_MicroServer
http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-4237916-4237918-4237917-4248009.html
http://forums.overclockers.com.au/showthread.php?t=905262

If 8TB is full, you need to stop the obsessive collection of warez/pr0n/torrentz you are never likely to watch again.

Re:you may want to have raid 5 or 6

[sirsnork]

Yup, start again.

Pick a board with plenty of SATA ports, put a modest amount of RAM and CPU in it. Make sure it's got PCI-E slots (what hasn't these days) and go from there.

Use bigger drives than you are currently, it's a bad time to buy drives so wait if you can but just build a new box from scratch and save yourself the headache of trying to migrate drives or retain data while upgrading drives one at a time in an existing array.

New machine, 3TB drives x as many as you want (6 would about double your capacity), add a 4 port PCI-E SATA card if you need it and rsync all the data across, job done

Hard drive math and hobbyist heuristics

[Wrath0fb0b]

Ballpark figures, this isn't exact, redo it with your preferred constants, I'm just trying to explain my reasoning against huge enclosures with > 10 drives,

Standard drive idle usage (W) ~ 10W [1]
Low-power (green) drive idle usage (W) ~ 5W [1]
Cost of power ~0.20 $/KWH
Cost of an older drive per year = $17
Cost of a green drive per year = $8.50
Replacing 6x500GB older drives with one 3TB green power savings = $95/yr

So think about that for a sec. At $150[2] for a 3TB drive, you cover the price in power savings in 18 months. That's assuming that there is zero fixed-cost per drive. At the point where you are talking about adding SATA controllers or fancy multi-bay enclosures or, worse, external enclosures with their own PSUs (and fans!), the turnaround-point for older drives is far sooner.

I'm a hobbyist, I understand that it's really cool to make do with older hardware and feel like you aren't letting anything to go waste but sometimes using old hardware instead of buying new is penny-wise and pound-foolish. Spending money on increasing how many hard drives you can accommodate instead of just buying newer high-capacity lower-wattage drives is absolutely batty; especially when you get into the price for anything remotely good in the RAID dept.

My advice, move everything to the largest capacity drives that are reasonably priced (after the flood damage is sorted). Replace the drives when you can do between 4:1 and 6:1 replacement -- should be every 3-4 years. Live happily, quietly and simpler. Save money on power transparently.

[1] http://hothardware.com/Reviews/Western-Digital-2TB-Caviar-Green-Power-Hard-Drive/
[2] I bought some Hitachi 3TBs before the Thailand floods at $130 on Newegg. Of course you would be silly as heck to buy hard drives now for your hobby storage project before they at least fall back to pre-flood level.
[3] http://www.newegg.com/Product/Product.aspx?Item=N82E16817182221
[4] Older drives need not go to waste, they can become offline storage with a simple USB dock[3] -- make a backup, throw it in an anti-static bag, leave it at your relative's house when you visit!
[5] http://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_time.svg

Re:Mac mini or apple Tv

[hairyfeet]

If he just wants to go cheap and its gonna be stuck in some garage anyway i'd say white trash it like we did at my last shop. Me and the boss took a couple of the biggest cheapo ATX cases we could get our hands on, took out the motherboard mount and cut the frame on one, and finally a couple of small weld stuck them together. We ended up with something like 16 SCSI drives in that sucker for a total of 2Tb when most folks were still getting 80Gb drives. with those babies loaded with every single driver for just about every piece of hardware up to that point it was quick work to reload a PC.

:As for the board I'd go with something like a cheap AMD board with a nice cheap Phenom low power quad. you can get a Phenom I quad for $55 at Starmicro and add a couple of Gb of DDR 2 for maybe $25, board for around $40 and finally a decent HSF for around $15.

So when you figure in the cases which with no PSU can be had for less than $20 a piece from many places like Geeks you are looking at a final total of around $185 for a quad with 2Gb of RAM with an add on card for adding more SATA slots. Sure it won't be the prettiest thing around but if its just gonna be serving files from a corner somewhere who cares? I'd add a little more and get a full size board with lots of SATA slots and 3 PCI slots for adding more SATA cards and you'd have a thing you could load to the brim and with the low power Phenom quad you'll have plenty of power for controlling the whole deal and maybe even it doing some of the transcoding work via scripts at night when its not serving files.

Re:Mac mini or apple Tv

[sortius_nod]

Just because you don't understand the request doesn't mean it's gobbledegooke, it means your knowledge is limited.

Personally, I don't trust any auto-encoding solutions as they easily go haywire. I'd suggest doing that all by hand.

Ext3 is fine, & a rack mount is a necessity. If you want smooth operation of the system, at least 2 network cards are a must (I run 3, 2 bonded for media/SMB, 1 for management & VPN). I'd suggest having a decent 16 port switch in the house & running the 2 (or more) cables to the box.

For DLNA I just run miniDLNA & for torrents I've just set up uTorrent with a web interface. There's very little my desktop actually does.

Re:Mac mini or apple Tv

[BLKMGK]

Ummmm, why two nic? The HDD and it's bus are the bottleneck - I cannot max out a gig-E nic now on my server but can play 1080P via x.264 with surround sound DTS encoded on a 100meg connection. No stuttering, no issues, no muss no fuss. It's wired for gig-E but sadly this nic refuses to synch at it.

I WOULD run some sort of redundant storage. However I wouldn't go traditional RAID. Striping data across disks along with parity makes for lots of speed, also means the disks never shut down. No thanks! I happen to use unRAID from Lime Technology. Parity isn't striped, it's held on a single disk. Each disk uses a standard format - ResierFS (ick). But's journaled and standard enough that recovery is easier. On top of that I can pull a disk and get data from it on another machine - one disk removed I still see all my data. Last but not least - if I lose multiple disks at once I only lose the data on THOSE disks and not the entire thing. Losing one disk I lose nothing. In 6+ years or so I've never lost more than one disk. Since parity isn't striped and neither is data one of my servers has ALL drives spun down and quiet, the other has just 2. Between them I have a bit over 22TB worth of disk BTW and not all are 2TB disks but as I fill smaller ones I swap in 2TB disks, the system migrates the data fine. Other programs can be run on this system, it's Linux based, but it's trickier than a full OS install - which has been done by some too.

Honestly to me it sounds like his current setup is working save for disk space. Why not just upgrade to 2TB disks? Surely he has more than 4 ports? If add-on cards are needed they are plentiful. I would also suggest using 4n1 cages for easy swapping. A low speed CPU is fine, underclock it maybe too - HDD bus is the bottleneck!

Re:Mac mini or apple Tv

[TheGratefulNet]

unraid sounds just like what I was *designing* for myself (doh!).

the idea of lots of disks always spinning is just moronic for home-users! we are NOT a data center! the game is different, here; and it has taken the industry a while to learn this.

I had a gazillion disks spinning in big-time home raid. for years. blech! lots of noise, heat, failure modes and like you said, you lose too much, you lose it all! the idea of being able to take a single spinner out, mount it on a 'dumb' system and read it, that's super powerful! don't discount that, folks! its worth its weight in gold.

my approach is to just be an idiot human (I do that part well) and save lots of copies of the files to lots of places. I do that anyway. then run a database job that will traverse your filesystem, get every file's size, date, md5 hash and any other tag info you want to peek inside, for. and run a smart differ on it. keep 'n' copies; report partial copies; trim the ones that are beyond what you need (if you want 3 copies, kill any 4,5,etc version you see). use ANY format you want for the disks, ntfs, ext3, jfs, whatever. each disk has a disk-id ('blkid' on linux shows this easily) and so each file has a disk-id to show where its located.

the final part of this is to keep all disks spun down and have the database know which disk the file is on (the one you want to watch/play) and have it do power-mgmt (as I call it) and spin the drive UP. either have it auto-spin down or spin down via mgmt when your 'session' is over.

spinning the disks is easy if they are all external esata/usb/fw. even simple x10 powerline relays can do this (my first proto will use this style of distrib tech). for internal drives, you power their molex's on/off via relays, via software control.

noise is stupid! power/heat wastage is stupid! raid is stupid for most of us. lets get beyond the gazillion spinners and be smarter about our large disk collections.

(oblig: I'm using arduinos and linux as the controller on all this)