Ask Slashdot: Best Kit For a Home Media Server?

First time accepted submitter parkejr writes "I started off building a media library a few years ago with an old PC running Ubuntu. Folders for photos, ogg vorbis music from my CD collection, and x264 encoded mkv movies. I have a high spec machine for encoding, but over the years I've moved the server to a bigger case, with 8 TB of disk capacity, and reverted back to Debian, but still running with the same AMD Sempron processor and 2GB RAM. It's working well, it's also the family mail server, and the kids are starting to use it for network storage, and it runs both link and twonkyserver, but my disks are almost full, and there are no more internal slots. The obvious option to me is to add in a couple of SATA PCI cards, to give me 4 more drives, and buy an externally powered enclosure, but that doesn't feel very elegant. I'm a bit of an amateur, so I'd like some advice. Should I start looking at a rack system? Something that can accommodate, say, 10 3.5" drives (I'm thinking long term, and some redundancy)? Also, what about location — I could run some cat6 to the garage and move it out of the house, in case noise is an issue. Finally, what about file format, file system, and OS/software? I'm currently running with ext3 and Debian Squeeze. Happy with my audio encoding choice, but not sure about x264 and mkv. I'd also consider different media server software, too. Any comments appreciated."

Comments

No reason to change from H.264

[InterestingFella]

Why would you change away from x264 and mkv. They are the industry standards. Not just in computers, but every way in the distribution chain. Going about it for some FOSS reason is just stupid because they're only for your own use, not for distribution. You would be either spending double the space or get half the quality by going with something other than H.264, and on top of that you introduce yourself additional problems because they are not what everyone uses.

Re:No reason to change from H.264

[Ltap]

Actually, Matroska has a number of good advantages:
1: More open than MP4 -- it has none of the ugly MPEG-LA overtones.
2: More codec support than MP4.
3: More consistency -- there aren't PS3-oriented versions or AppleTV versions of Matroska. MP4's device support might be wide, but when every player and device seems to have its own version with its own ridiculous, restrictive standards, it doesn't really qualify as supported at all.
4: Content management: Matroska is the easiest to mux and makes it great to work with alternate audio and video streams, subtitles, etc. You get the widest range of options of any container out there.

While Matroska might be used by some because it is perceived as more "elite", there are many very valid reasons for using it, mainly revolving around flexibility and openness. As someone who has worked with both containers, I can say that Matroska far more easily delivers what I want and the tools for working with it are generally free (in both senses), more usable, and more powerful.

Re:No reason to change from H.264

[Stiletto]

Ditch .mkv as soon as possible. It's an almost completely unsupported container. Even among software that supposedly supports it there can be compatibility issues. It's popular in the ripping/pirate communities precisely because it's a pain to use. Just getting your videos to work on a regular basis is a mark of distinction.

I disagree. Having just converted my entire collection over to mkv, I'm never looking back. There are some great reasons to use it:

1. I've found MKV to have better support for chapters
2. MKV has heaps better support for subtitles (I could never manage to get subtitles to properly work across players using MP4)
3. MKV can hold just about any video and audio encoding, not just H.263 and H.264
4. (If you care) MP4 has patent issues, whereas MKV does not

Re:No reason to change from H.264

[Telvin_3d]

1) I've never had problems with using Handbreak for chapters. But one anecdote is about as useless as another for this kind of thing.
2) Only system I've ever found that supports .mp4 but not the subtitles is the PS3, and all indications are that it was deliberate. So fuck Sony and the horse they rode in on (slung underneath). On the other hand, PS3 doesn't support mkv at all

3) This is not a good thing if your goal is to play back the content on any system but the one it was made on. Ever gone internet hunting for that one weird codec that you used for a few months a couple years ago? No? Me neither because I'm not dumb enough to think that 'can jam anything into it' is a good thing in a media format.

4) This is... debatable. Both formats are open standards and open source. You can look at the specs and the code for either. The patents for .mp4 are known and need to be licensed if you are a large commercial operation. The patents for mkv are god-knows-what and may or may not get eaten alive the first time that the patents ever become important. No one knows. Pick your poison.

Re:No reason to change from H.264

[Telvin_3d]

While Handbrake is an awesome piece of software, it's not a remuxer. It doesn't support passthrough for the video streams. They will always be re-compressed. Often with little or no loss of visible quality, but some loss will occur.

Re:No reason to change from H.264

[Ltap]

I find it amazing that you are even trying to argue that "fewer supported codecs" is a good thing. MKV isn't MKV isn't MKV -- it's all about the codecs you use, and that's what it should be about. The container is a way of gathering together media streams, but doing something like designing a player and assuming that, for a given container, there will be only a single or a couple of codecs (when that container actually supports many) is ridiculous. If your player can't examine a container it theoretically supports and find out which codecs it uses, then you shouldn't use that player because its manufacturers made far too many underlying assumptions. If you are arguing that MKV allowing unusual or poor codec choices means people will make them, then yes -- some people will, for various reasons. But does that mean people shouldn't be allowed to make a decision just because you don't think it's a good idea, based on your usage case? If there's anything FOSS has taught me, it's that many people will have truly unique circumstances which require very specific things to happen to reach a proper solution and flexibility is everything. What you seem to be arguing is that no one should be allowed to use a hypothetical ideal container in ways you don't want because you consider those improper.

Device support might be important to you, but I've just never understood why people contort themselves into making decision which are, frankly, idiotic simply because they have a crappy DVD player or something which only supports a few things in a handful of configurations. The most sensible and flexible solution is ... tada! A PC hooked up to a television with a cable, where the television is just yet another display device, no different from a monitor. Trying to wrestle with a DVD or (ugh) Blu-Ray player is just asking for some manufacturer to screw you over.

What I care about most is media support. What can you fit in? MKV's subtitle handling is simply the best around bar none. You get complete flexibility to store hardcoded or softcoded subtitles in any way you want and you can easily do the same with audio, something which cannot be said for most containers. Unless you are trying to square the circle by wrestling with uncooperative devices you are unwilling to give up, there's really no reason to use anything else, at least in the present.

Re:No reason to change from H.264

[Osty]

Others have touched on your other points, so I wanted to address this:

Switch to a standard .mp4 container. Much better supported on hardware or software. Some day you will want to be able to stream from your server to a thin set-top box or load a file on your kid's phone. On that day .mkv will make you cry.

Plenty of thin set-top box clients play mkvs already. Devices from Western Digital (WD TV Live), Netgear, Seagate, Roku, Popcorn Hour, Boxee, and many others all support mkv out of the box, with header compression support, subtitles, chapters, multiple audio and video streams, and some even support 3D (not the new mk3d format yet, but SBS works) and will play subtitles correctly. Most mkv files contain MPEG2, h264, or VC-1 video and AC3 or DTS (or the newer Dolby TrueHD, DTS-HD, etc) and all of these players handle those just fine. Don't blame the container for being flexible enough to allow any random codec. Blame whatever crap source you stole your videos from for using a random codec. Those of us archiving our DVDs and Blu-Rays will either encode in H264 or remux the original MPEG2/H264/VC-1 streams untouched and have no problems.

If you want to move videos to a phone, that's easy enough to do. The beauty of being a completely open container format means that it's trivial to demux Matroska containers into their component streams, which than then be remuxed into mp4 for devices that suck. Since you'll probably want to down-res the videos anyway for handheld formats (even on tablets you won't want higher than 720p), there's no reason to keep the originals in mp4. Keep your original, untouched videos in mkv and re-encode at lower resolution and bitrates into mp4 using Handrake for mobile devices.

Divx (yeah, whatever, they're still relevant) has adopted MKV as their HD container format, and the proliferation of "networked media tank" devices plus Matroska's openness makes it not only relevant but desirable for long-term video storage. Using 5+ year old devices like Xbox 360 and PS3 as your benchmark for what containers to use would be a bad idea (Xbox still doesn't even support 6-channel AAC in mp4, never mind supporting AC3).

Arduino.

[Anonymous Coward]

Just kidding.

Larger disks

[Anonymous Coward]

Larger disks. 4 TB should be available very soon (maybe now?).
http://www.extremetech.com/computing/108665-hitachi-ships-worlds-first-4tb-hard-drive-sticks-it-to-thor

HP Microserver

[quenda]

For a slightly more sane solution than rackmounting at home, consider the HP microserver.
Very low power (12W CPU), small, quiet, cheap, server grade, no Windows tax, holds four pluggable 3.5" drives plus optical (which some people swap for a 5th HDD for RAID5.)

http://blog.thestateofme.com/2011/05/14/review-hp-microserver/
http://www.silentpcreview.com/HP_Proliant_MicroServer
http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-4237916-4237918-4237917-4248009.html
http://forums.overclockers.com.au/showthread.php?t=905262

If 8TB is full, you need to stop the obsessive collection of warez/pr0n/torrentz you are never likely to watch again.

Re:HP Microserver

[EdIII]

If 8TB is full, you need to stop the obsessive collection of warez/pr0n/torrentz you are never likely to watch again.

As opposed to the obsessive collecting of physical media that can be scratched and takes up 10x+ the space?

My family since the 80s has amassed literally 10k CD/DVDs as well as almost 100 laserdisc titles. Not to mention a buttload of VHS tapes that we offloaded years ago.

It has all been converted to digital storage. Since it is on multiple RAID 5 devices and I run a cron job that checks the MD5 sigs against a database I know that it is in good condition.

Of course this requires constant rollover of the data from hard drive to hard drive. Half the drives have failed over the years and it has moved between multiple NAS systems. We still have all the data.

In addition to that, we have over 100k family photos collected from all of our relatives scanned and tagged as well.

Our collection is nearing 20 TB. With the low cost of drives we have backups in lead lined containers in safety deposit boxes at two banks. We swap them out every year or so adding to it. I am really looking forward to long term archival storage that is write once and designed to last 100 years plus. I'll pay for that.

Now I know you may be thinking obsession, but we *paid* for it. Paying twice for music or movies is just plain insane and we never fell for the HD/Bluesuck shit they were shoving down our throats. Well my parents did, but Spiderman solved that problem the first time it could not be played because the encryption changed. Since then they are back on DVD only and we are waiting for a HD storage method that does not involve constant Big Brother monitoring and DRM in our houses.

Then there is the most obvious benefit of all. You only have to rip the music or movie one time. Been years since we bought an actual CD, but you get my point.

The convenience of having all of your media at your fingertips without touching physical media is pretty damn nice.

Guess how much storage space you need for thousands of DVD/CDs when they are packed into spindles and put into storage? A heck of lot less than you would expect. Fits in a closet.

Re:Mac mini or apple Tv

[Kagetsuki]

I'm sorry, did you read the summary? Do you honestly think a Mac mini is a step up from that or solves the problems presented?

Re:you may want to have raid 5 or 6

[sirsnork]

Yup, start again.

Pick a board with plenty of SATA ports, put a modest amount of RAM and CPU in it. Make sure it's got PCI-E slots (what hasn't these days) and go from there.

Use bigger drives than you are currently, it's a bad time to buy drives so wait if you can but just build a new box from scratch and save yourself the headache of trying to migrate drives or retain data while upgrading drives one at a time in an existing array.

New machine, 3TB drives x as many as you want (6 would about double your capacity), add a 4 port PCI-E SATA card if you need it and rsync all the data across, job done

Re:Mac mini or apple Tv

[pcx]

A mac mini is a perfectly valid (if expensive) media solution. I personally use 2 synology NASs feeding a Mac Mini which is a dedicated iTunes server feeding 4 apple TVs (When I win the lotto I might try out a promise pegasus but until then the synology NASs are growing the library nicely). It's a bit of a pain to remux mkvs to mp4s but it works and it's a really nice solution once you've gone through the headaches of setting it up (yes better than a boxee box -- tried it, now gathering dust, better than every DLNA solution I've tried). The Apple TVs are just really, really nice media end-points and iTunes is a perfectly good management system.

But yea iCloud has no place in any video solution (your purchased shows can be streamed -- movies can not, and this solution will introduce you to your ISP's bandwidth caps very quickly).

Hard drive math and hobbyist heuristics

[Wrath0fb0b]

Ballpark figures, this isn't exact, redo it with your preferred constants, I'm just trying to explain my reasoning against huge enclosures with > 10 drives,

Standard drive idle usage (W) ~ 10W [1]
Low-power (green) drive idle usage (W) ~ 5W [1]
Cost of power ~0.20 $/KWH
Cost of an older drive per year = $17
Cost of a green drive per year = $8.50
Replacing 6x500GB older drives with one 3TB green power savings = $95/yr

So think about that for a sec. At $150[2] for a 3TB drive, you cover the price in power savings in 18 months. That's assuming that there is zero fixed-cost per drive. At the point where you are talking about adding SATA controllers or fancy multi-bay enclosures or, worse, external enclosures with their own PSUs (and fans!), the turnaround-point for older drives is far sooner.

I'm a hobbyist, I understand that it's really cool to make do with older hardware and feel like you aren't letting anything to go waste but sometimes using old hardware instead of buying new is penny-wise and pound-foolish. Spending money on increasing how many hard drives you can accommodate instead of just buying newer high-capacity lower-wattage drives is absolutely batty; especially when you get into the price for anything remotely good in the RAID dept.

My advice, move everything to the largest capacity drives that are reasonably priced (after the flood damage is sorted). Replace the drives when you can do between 4:1 and 6:1 replacement -- should be every 3-4 years. Live happily, quietly and simpler. Save money on power transparently.

[1] http://hothardware.com/Reviews/Western-Digital-2TB-Caviar-Green-Power-Hard-Drive/
[2] I bought some Hitachi 3TBs before the Thailand floods at $130 on Newegg. Of course you would be silly as heck to buy hard drives now for your hobby storage project before they at least fall back to pre-flood level.
[3] http://www.newegg.com/Product/Product.aspx?Item=N82E16817182221
[4] Older drives need not go to waste, they can become offline storage with a simple USB dock[3] -- make a backup, throw it in an anti-static bag, leave it at your relative's house when you visit!
[5] http://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_time.svg

Re:Mac mini or apple Tv

[hairyfeet]

If he just wants to go cheap and its gonna be stuck in some garage anyway i'd say white trash it like we did at my last shop. Me and the boss took a couple of the biggest cheapo ATX cases we could get our hands on, took out the motherboard mount and cut the frame on one, and finally a couple of small weld stuck them together. We ended up with something like 16 SCSI drives in that sucker for a total of 2Tb when most folks were still getting 80Gb drives. with those babies loaded with every single driver for just about every piece of hardware up to that point it was quick work to reload a PC.

:As for the board I'd go with something like a cheap AMD board with a nice cheap Phenom low power quad. you can get a Phenom I quad for $55 at Starmicro and add a couple of Gb of DDR 2 for maybe $25, board for around $40 and finally a decent HSF for around $15.

So when you figure in the cases which with no PSU can be had for less than $20 a piece from many places like Geeks you are looking at a final total of around $185 for a quad with 2Gb of RAM with an add on card for adding more SATA slots. Sure it won't be the prettiest thing around but if its just gonna be serving files from a corner somewhere who cares? I'd add a little more and get a full size board with lots of SATA slots and 3 PCI slots for adding more SATA cards and you'd have a thing you could load to the brim and with the low power Phenom quad you'll have plenty of power for controlling the whole deal and maybe even it doing some of the transcoding work via scripts at night when its not serving files.

Re:I was saying maybe a raid card but non raid car

[nabsltd]

useing software raid is ok most boards have about 6 ports so if you want like 10 then maybe a x4 or better pci-e card may be needed.

Or, get an actual server board (this is gonna be a server, right?), like this one. That's six SATA ports and 8 SAS ports. If you flash the SAS ROM to the "no-RAID" version, the controller is recognized natively by Linux. In addition, you get lots of PCIe connectivity, a pair of Gigabit Ethernet ports, and IPMI (allowing remote power cycle).

Then, find a full-tower case with lots of 5-1/4" drive bays, and add hot swap bays. There are smaller versions, as well...just budget what you need for drives.

I use the motherboard I referenced along with an add-on 8-port SATA card (anything supported by Linux would be fine) and two of the drive bays for ten 2TB drives in RAID-10. I boot Fedora off a pair of SSDs in RAID-1 and also have four 2-1/2" 750GB drives in RAID-10. The 10TB array serves iSCSI over 10Gbit Ethernet to ESX systems that hold all my VMs, with the 1.5TB array as local and NFS storage. There's still PCIe slots available if you need more controller cards.

With this setup, the VMs are how everything is accessed, so you can pick whatever OS you want to face client machines.

Thoughts from my home storage server experience.

[jafo]

I wrote about the latest storage server I built back in 2008, and a lot of my thoughts at the time are written up in http://www.tummy.com/Community/Articles/ultimatestorage2008/

However, to answer a few of your questions...

External disc enclosures? Avoid them like the plague. My initial experience with the 5 bay eSATA enclosures was pretty good -- sometimes it wouldn't pick up the external drives, but usually I could get it to find them after some tweaking, rebooting, etc... I ended up getting 3 of them, the AMS DS-2350S, which at the time were well reviewed, etc... I have since pulled all 3 of them out of active use and have them just sitting around. I don't know exactly the mode of the failures, but eventually after replacing some with others, I finally put them in internal SATA enclosures, which have been very reliable (I used these Supermicro CSE-M35T-1.

Also note that eSATA connectors don't really hold on that well. If anything, they're not as robust as internal SATA connectors, despite being outside the case where they can get banged around.

If I were to do it over again, I'd probably stick with the case I started with, with 5 internal 3.5" bays, and 3 front 5.25" bays, and put the Supermicro in there. I'd also probably go with fewer big drives rather than more smaller drives like I did previously (even though at the time the drives were free, I had them from another project).

As far as running it in the garage, don't even think it, unless your garage is not where you store your cars. I have some computers that I've run in the garage for the last 9 months, and they are filthy, I've had a lot of fan failures, lots of dust, insects, and random other crap. I put mine in our furnace room, which has enough extra space.

As far as using a server case? Hard to see the payback there unless you have a cabinet. Most server cases are HUGE, heavy, and expensive. A 3U case with 12 drive bays likely costs $500, plus you usually have to deal with special form-factor power supplies, expect to spend another $200 on one of those. I wouldn't do it, and I have a 3U 12-bay Chenbro case just sitting at my office that I could re-purpose.

As far as the file-system, I selected ZFS (via zfs-fuse under Linux) and I've been VERY happy with it. The primary benefit is that it checksums *ALL* data and can recover from some types of corruption or at least alert about corruption if it can't correct it. So, if you are storing photos or home videos that you may not be accessing very often, that's good peace of mind to have, I know in 10 years I won't go to look at some photographs I've taken and find they were silently corrupted. Of course, you could get similar benefits by saving off a database of file checksums and checking and alerting if they are bad. Really the only downside of ZFS that I've seen is that if you need to do a RAID rebuild it is a seek-heavy task rather than just streaming. I have a 8x2TB drive array that I'm currently rebuilding (drive failure, at work), and it's 33% done after 31 hours. A normal RAID-5 array would have rebuilt that in what, 10? The system is idle except for the rebuild.

If you care about the data going into it, make sure you checksum and verify the files regularly.

The 8 port PCI SATA card I got is fantastic, it's a Supermicro with the Marvel chipset and is very well supported (even supported by Nexenta).

Finally, all this data is encrypted, so if someone were to burgle us I only have to worry about them getting the hardware, I don't have to worry about them now having scanned bills and other documents and other personal and private data, etc... This is why I'm running ZFS in Linux, it gave me encryption plus ZFS (not available otherwise in 2008), as well as being an OS I'm very familiar with.

As far as OS, I am personally running CentOS on my system because that means I can install and set it up and then forget about it for quite a few years, except for regularly running "yum update". Debian should be fine, but you will get/have to track upstream changes more frequently.

Re:Mac mini or apple Tv

[sortius_nod]

Just because you don't understand the request doesn't mean it's gobbledegooke, it means your knowledge is limited.

Personally, I don't trust any auto-encoding solutions as they easily go haywire. I'd suggest doing that all by hand.

Ext3 is fine, & a rack mount is a necessity. If you want smooth operation of the system, at least 2 network cards are a must (I run 3, 2 bonded for media/SMB, 1 for management & VPN). I'd suggest having a decent 16 port switch in the house & running the 2 (or more) cables to the box.

For DLNA I just run miniDLNA & for torrents I've just set up uTorrent with a web interface. There's very little my desktop actually does.

Re:Mac mini or apple Tv

[BLKMGK]

Ummmm, why two nic? The HDD and it's bus are the bottleneck - I cannot max out a gig-E nic now on my server but can play 1080P via x.264 with surround sound DTS encoded on a 100meg connection. No stuttering, no issues, no muss no fuss. It's wired for gig-E but sadly this nic refuses to synch at it.

I WOULD run some sort of redundant storage. However I wouldn't go traditional RAID. Striping data across disks along with parity makes for lots of speed, also means the disks never shut down. No thanks! I happen to use unRAID from Lime Technology. Parity isn't striped, it's held on a single disk. Each disk uses a standard format - ResierFS (ick). But's journaled and standard enough that recovery is easier. On top of that I can pull a disk and get data from it on another machine - one disk removed I still see all my data. Last but not least - if I lose multiple disks at once I only lose the data on THOSE disks and not the entire thing. Losing one disk I lose nothing. In 6+ years or so I've never lost more than one disk. Since parity isn't striped and neither is data one of my servers has ALL drives spun down and quiet, the other has just 2. Between them I have a bit over 22TB worth of disk BTW and not all are 2TB disks but as I fill smaller ones I swap in 2TB disks, the system migrates the data fine. Other programs can be run on this system, it's Linux based, but it's trickier than a full OS install - which has been done by some too.

Honestly to me it sounds like his current setup is working save for disk space. Why not just upgrade to 2TB disks? Surely he has more than 4 ports? If add-on cards are needed they are plentiful. I would also suggest using 4n1 cages for easy swapping. A low speed CPU is fine, underclock it maybe too - HDD bus is the bottleneck!

Re:Mac mini or apple Tv

[iamhassi]

He wants a media server not a HTPC.

then pretty much any old PC with a bunch of 3.5" drives would work.

I googled "build a media server" and found several guides. Here's one for $300 with six 3.5" bays and here's a 4U rackmount server case with twenty 3.5" drive bays for roughly $1,000

Re:Mac mini or apple Tv

[TheGratefulNet]

unraid sounds just like what I was *designing* for myself (doh!).

the idea of lots of disks always spinning is just moronic for home-users! we are NOT a data center! the game is different, here; and it has taken the industry a while to learn this.

I had a gazillion disks spinning in big-time home raid. for years. blech! lots of noise, heat, failure modes and like you said, you lose too much, you lose it all! the idea of being able to take a single spinner out, mount it on a 'dumb' system and read it, that's super powerful! don't discount that, folks! its worth its weight in gold.

my approach is to just be an idiot human (I do that part well) and save lots of copies of the files to lots of places. I do that anyway. then run a database job that will traverse your filesystem, get every file's size, date, md5 hash and any other tag info you want to peek inside, for. and run a smart differ on it. keep 'n' copies; report partial copies; trim the ones that are beyond what you need (if you want 3 copies, kill any 4,5,etc version you see). use ANY format you want for the disks, ntfs, ext3, jfs, whatever. each disk has a disk-id ('blkid' on linux shows this easily) and so each file has a disk-id to show where its located.

the final part of this is to keep all disks spun down and have the database know which disk the file is on (the one you want to watch/play) and have it do power-mgmt (as I call it) and spin the drive UP. either have it auto-spin down or spin down via mgmt when your 'session' is over.

spinning the disks is easy if they are all external esata/usb/fw. even simple x10 powerline relays can do this (my first proto will use this style of distrib tech). for internal drives, you power their molex's on/off via relays, via software control.

noise is stupid! power/heat wastage is stupid! raid is stupid for most of us. lets get beyond the gazillion spinners and be smarter about our large disk collections.

(oblig: I'm using arduinos and linux as the controller on all this)