Ask Slashdot: Best Kit For a Home Media Server?

First time accepted submitter parkejr writes "I started off building a media library a few years ago with an old PC running Ubuntu. Folders for photos, ogg vorbis music from my CD collection, and x264 encoded mkv movies. I have a high spec machine for encoding, but over the years I've moved the server to a bigger case, with 8 TB of disk capacity, and reverted back to Debian, but still running with the same AMD Sempron processor and 2GB RAM. It's working well, it's also the family mail server, and the kids are starting to use it for network storage, and it runs both link and twonkyserver, but my disks are almost full, and there are no more internal slots. The obvious option to me is to add in a couple of SATA PCI cards, to give me 4 more drives, and buy an externally powered enclosure, but that doesn't feel very elegant. I'm a bit of an amateur, so I'd like some advice. Should I start looking at a rack system? Something that can accommodate, say, 10 3.5" drives (I'm thinking long term, and some redundancy)? Also, what about location — I could run some cat6 to the garage and move it out of the house, in case noise is an issue. Finally, what about file format, file system, and OS/software? I'm currently running with ext3 and Debian Squeeze. Happy with my audio encoding choice, but not sure about x264 and mkv. I'd also consider different media server software, too. Any comments appreciated."

Comments

No reason to change from H.264

[InterestingFella]

Why would you change away from x264 and mkv. They are the industry standards. Not just in computers, but every way in the distribution chain. Going about it for some FOSS reason is just stupid because they're only for your own use, not for distribution. You would be either spending double the space or get half the quality by going with something other than H.264, and on top of that you introduce yourself additional problems because they are not what everyone uses.

Re:No reason to change from H.264

[Telvin_3d]

Why would you change away from x264 and mkv. They are the industry standards.

Stick with x264. It is open source and industry stanard. Not open licensed from a commercial point of view, but all the software side and standards are completely open.

Ditch .mkv as soon as possible. It's an almost completely unsupported container. Even among software that supposedly supports it there can be compatibility issues. It's popular in the ripping/pirate communities precisely because it's a pain to use. Just getting your videos to work on a regular basis is a mark of distinction.

Switch to a standard .mp4 container. Much better supported on hardware or software. Some day you will want to be able to stream from your server to a thin set-top box or load a file on your kid's phone. On that day .mkv will make you cry.

There are server remuxers floating around that repackage the video and audio streams into a new container. No re-compression or quality loss. You can fold any subtitle files or other extras into the file at the same time.

Re:No reason to change from H.264

[Ltap]

Actually, Matroska has a number of good advantages:
1: More open than MP4 -- it has none of the ugly MPEG-LA overtones.
2: More codec support than MP4.
3: More consistency -- there aren't PS3-oriented versions or AppleTV versions of Matroska. MP4's device support might be wide, but when every player and device seems to have its own version with its own ridiculous, restrictive standards, it doesn't really qualify as supported at all.
4: Content management: Matroska is the easiest to mux and makes it great to work with alternate audio and video streams, subtitles, etc. You get the widest range of options of any container out there.

While Matroska might be used by some because it is perceived as more "elite", there are many very valid reasons for using it, mainly revolving around flexibility and openness. As someone who has worked with both containers, I can say that Matroska far more easily delivers what I want and the tools for working with it are generally free (in both senses), more usable, and more powerful.

Re:No reason to change from H.264

[Stiletto]

Ditch .mkv as soon as possible. It's an almost completely unsupported container. Even among software that supposedly supports it there can be compatibility issues. It's popular in the ripping/pirate communities precisely because it's a pain to use. Just getting your videos to work on a regular basis is a mark of distinction.

I disagree. Having just converted my entire collection over to mkv, I'm never looking back. There are some great reasons to use it:

1. I've found MKV to have better support for chapters
2. MKV has heaps better support for subtitles (I could never manage to get subtitles to properly work across players using MP4)
3. MKV can hold just about any video and audio encoding, not just H.263 and H.264
4. (If you care) MP4 has patent issues, whereas MKV does not

Re:No reason to change from H.264

[kyrio]

MKV is used because it's currently the best container, there's nothing else to it. With the dumb shit you are saying, you obviously have no clue about the state of the technology.

Re:No reason to change from H.264

[Anonymous Coward]

Although I do agree that using x264 in mp4 is a good idea, the fact is that MKV (Matroska) is a very well accepted/supported container. You must feel like a complete idiot for claiming that is not.

The issue with some MKV files is that they include UNSUPPORTED video / sound formats. (in case poeple didn't know, MKV is not a video/audio format ... it is a container ... a kind of zip for media files). For example a lot of completely delusional people love to use Ogg Vorbis as sound, which is barely supported outside of a few open source players and even there ... they don't work that well. Then a few crazy people like to mix WMV with Ogg .... and you get garbage.

There is nothing wrong with MKV ... except for "encoders" who can't follow common sense rules.

And about finding a remuxer ..... I would suggest Handbrake (http://handbrake.fr/). It is the best mufti-platform tool ... and it is free. Also, it automatically converts ogg sound into AC3 without losing audio sync (which is a problem with many other tools).

Re:No reason to change from H.264

[Telvin_3d]

1) This is debatable, but not really relevant for the discussion. Depends on how you judge the difference between known licenses and god-knows-what patent issues.

2) Not sure if this counts as a positive feature. AVIs are famous for the ten-million random codecs they might contain. Just because you can get the container to wrap around a particular stream doesn't mean you will ever be able to play it back again.

3) Those 'versions' are actually a feature of the h264 codec, not the container. You get the same thing on mp4 or mkv. You may not have realized it because it's just assumed that mkv is incompatible ;)

It's a combination of the supported profiles/levels. Basically rising levels of extra quality in exchange for higher hardware requirements. They are fully forward-compatible, so anything compressed with the standard profiles will play on any device that supports higher profiles, but something that has been recorded specifically for a high-end system will not play on a low-end one. Even phones are now starting to support level 5.1 high profile, which is the highest, so in another year or so the entire issue will disappear.

4) Again, I'm not sure this is a feature. Being able to jam anything you can dream of into a container is a poor choice if the goal is to play it back on any system but the one that was used to make the file.

Re:No reason to change from H.264

[Telvin_3d]

1) I've never had problems with using Handbreak for chapters. But one anecdote is about as useless as another for this kind of thing.
2) Only system I've ever found that supports .mp4 but not the subtitles is the PS3, and all indications are that it was deliberate. So fuck Sony and the horse they rode in on (slung underneath). On the other hand, PS3 doesn't support mkv at all

3) This is not a good thing if your goal is to play back the content on any system but the one it was made on. Ever gone internet hunting for that one weird codec that you used for a few months a couple years ago? No? Me neither because I'm not dumb enough to think that 'can jam anything into it' is a good thing in a media format.

4) This is... debatable. Both formats are open standards and open source. You can look at the specs and the code for either. The patents for .mp4 are known and need to be licensed if you are a large commercial operation. The patents for mkv are god-knows-what and may or may not get eaten alive the first time that the patents ever become important. No one knows. Pick your poison.

Re:No reason to change from H.264

[Telvin_3d]

While Handbrake is an awesome piece of software, it's not a remuxer. It doesn't support passthrough for the video streams. They will always be re-compressed. Often with little or no loss of visible quality, but some loss will occur.

Re:No reason to change from H.264

[Ltap]

I find it amazing that you are even trying to argue that "fewer supported codecs" is a good thing. MKV isn't MKV isn't MKV -- it's all about the codecs you use, and that's what it should be about. The container is a way of gathering together media streams, but doing something like designing a player and assuming that, for a given container, there will be only a single or a couple of codecs (when that container actually supports many) is ridiculous. If your player can't examine a container it theoretically supports and find out which codecs it uses, then you shouldn't use that player because its manufacturers made far too many underlying assumptions. If you are arguing that MKV allowing unusual or poor codec choices means people will make them, then yes -- some people will, for various reasons. But does that mean people shouldn't be allowed to make a decision just because you don't think it's a good idea, based on your usage case? If there's anything FOSS has taught me, it's that many people will have truly unique circumstances which require very specific things to happen to reach a proper solution and flexibility is everything. What you seem to be arguing is that no one should be allowed to use a hypothetical ideal container in ways you don't want because you consider those improper.

Device support might be important to you, but I've just never understood why people contort themselves into making decision which are, frankly, idiotic simply because they have a crappy DVD player or something which only supports a few things in a handful of configurations. The most sensible and flexible solution is ... tada! A PC hooked up to a television with a cable, where the television is just yet another display device, no different from a monitor. Trying to wrestle with a DVD or (ugh) Blu-Ray player is just asking for some manufacturer to screw you over.

What I care about most is media support. What can you fit in? MKV's subtitle handling is simply the best around bar none. You get complete flexibility to store hardcoded or softcoded subtitles in any way you want and you can easily do the same with audio, something which cannot be said for most containers. Unless you are trying to square the circle by wrestling with uncooperative devices you are unwilling to give up, there's really no reason to use anything else, at least in the present.

Re:No reason to change from H.264

[Osty]

Others have touched on your other points, so I wanted to address this:

Switch to a standard .mp4 container. Much better supported on hardware or software. Some day you will want to be able to stream from your server to a thin set-top box or load a file on your kid's phone. On that day .mkv will make you cry.

Plenty of thin set-top box clients play mkvs already. Devices from Western Digital (WD TV Live), Netgear, Seagate, Roku, Popcorn Hour, Boxee, and many others all support mkv out of the box, with header compression support, subtitles, chapters, multiple audio and video streams, and some even support 3D (not the new mk3d format yet, but SBS works) and will play subtitles correctly. Most mkv files contain MPEG2, h264, or VC-1 video and AC3 or DTS (or the newer Dolby TrueHD, DTS-HD, etc) and all of these players handle those just fine. Don't blame the container for being flexible enough to allow any random codec. Blame whatever crap source you stole your videos from for using a random codec. Those of us archiving our DVDs and Blu-Rays will either encode in H264 or remux the original MPEG2/H264/VC-1 streams untouched and have no problems.

If you want to move videos to a phone, that's easy enough to do. The beauty of being a completely open container format means that it's trivial to demux Matroska containers into their component streams, which than then be remuxed into mp4 for devices that suck. Since you'll probably want to down-res the videos anyway for handheld formats (even on tablets you won't want higher than 720p), there's no reason to keep the originals in mp4. Keep your original, untouched videos in mkv and re-encode at lower resolution and bitrates into mp4 using Handrake for mobile devices.

Divx (yeah, whatever, they're still relevant) has adopted MKV as their HD container format, and the proliferation of "networked media tank" devices plus Matroska's openness makes it not only relevant but desirable for long-term video storage. Using 5+ year old devices like Xbox 360 and PS3 as your benchmark for what containers to use would be a bad idea (Xbox still doesn't even support 6-channel AAC in mp4, never mind supporting AC3).

Re:No reason to change from H.264

[ararara_]

3) This is not a good thing if your goal is to play back the content on any system but the one it was made on. Ever gone internet hunting for that one weird codec that you used for a few months a couple years ago? No? Me neither because I'm not dumb enough to think that 'can jam anything into it' is a good thing in a media format.

Just because you can put anything in it doesn't mean that you should. Nor does it mean that you should go with anything that isn't standard. Using it over the mp4 container, does future proof it somewhat, as (the way I see it) h264 is likely to get depreciated before the matroska container (assuming adding support doesn't create issues, I'm not that savvy in regards to matroska specifically). It also allows you to provide video in using multiple formats without having to worry about using a different container for each one. Just plug in a different video stream and away you go. I'm not sure the OP will need too, but this could be handy depending on what they are doing with their video down the track.

Re:No reason to change from H.264

[Hognoxious]

The original claim wasn't that it's better, it was that it's an industry standard.

I own more devices that will play avi but not mkv than the other way round. Maybe it's just because it's newer, but it doesn't seem very standard to me.

Re:No reason to change from H.264

[Sentry23]

On 2) There are tons of devices that don't support MP4 substitles. (including the Xbox 360)
Why turn this into an anti Sony rant?

Re:No reason to change from H.264

[hairyfeet]

DivX 5 AVIs will play on damned near anything from that $30 Walamrt STB on up, you can get some pretty damned decent file sizes while having great picture and they are really easy to make with a wealth of converters out there.

Now as I said on my other post the AMD Phenom I quad CULV I linked to is only $55 and AMD AM2+ boards are dirt cheap now so that would take care of the hardware and if it were me and I were setting it up I'd use an old XP license sitting around to legit a copy of "XP Tiny Edition" that runs in just 56Mb of RAM (or if you don't give a shit about licenses they also have Tiny 2K3 but it'll take you a while to find a copy as the server version isn't as popular as Tiny XP and 7) and slap on Virtualdub and just have it convert in batches when the machine wasn't in use.

With the above parts and software you'd have a cheap quad server with tons of space that'll convert your videos to one of the most popular and widely supported formats easy peasy. Hell there are several tutorials on how to change aspect ratio to whatever your native is via VDub if you want to go to the extra trouble, me I have the VDub settings for 4x3 saved as a file for when my mom wants movies so she can watch them on her big old set (which she refuses to let me replace because "It works just fine for me") and have my Phenom II 6 core whip them off while i snooze. Can't get easier to use frankly.

Re:No reason to change from H.264

[Anonymous Coward]

Is MKV shit, or is the example library shit? There's a big difference there, but I think it's too "subtle" for your feeble mind.

Re:No reason to change from H.264

[GNious]

The container is a way of gathering together media streams, but doing something like designing a player and assuming that, for a given container, there will be only a single or a couple of codecs (when that container actually supports many) is ridiculous..

See: Samsung SmartTV and related equipment.
(Plays DVDs, but doesn't like MPEG2 in a transport-stream..? Only support H264 when using MKV.)

The most sensible and flexible solution is ... tada! A PC hooked up to a television with a cable, where the television is just yet another display device, no different from a monitor. Trying to wrestle with a DVD or (ugh) Blu-Ray player is just asking for some manufacturer to screw you over.

Have not seen a system that can take multiple HDMI inputs, play Blu-Ray/DVD, support DLNA and various containers+codec, handles DVB sources, support inet-tv, based on a PC. Have tried XBMC (yrch), Windows Media Center (double-yrch), and Front Row (ehhh?). As far as I can tell, all options are inflexible, incompetent and irritatingly crappy.

Re:No reason to change from H.264

[Anonymous Coward]

I'm fairly sure Roku *just* added MKV support like a month or two ago, and only on their newer Roku2. As is always the case with Roku, while it may work for some that does not mean it will work for everyone.

Re:No reason to change from H.264

[spire3661]

Who supports MKV commercially? What name brand consumer device that isnt a PC/MAC can i go to BestBuy and purchase that will play MKV?

Re:No reason to change from H.264

[Ltap]

I use VLC, MPC-HC and mplayer. What one does poorly, the other will do well. While Blu-Ray support for PCs is patchy at best, this is by design -- they want you to buy the player and be slave to them. As for DLNA, it's really only significant for specialized devices -- if you wanted to split things up between a few computers, all you would need would be a LAN and a bit of scripting.

It's just the wrong approach to immediately run out to a big "media system" program (especially WMC) when most of what you want can be handled much more simply by a basic media player (which is really what these are, only with a clunky interface).

Re:No reason to change from H.264

[cynyr]

At the end of the day, my "HTPC" is my ps3. So unless I want to demux and then remux every mkv on the fly for my PS3 I'll stick to my MP4s.

Re:No reason to change from H.264

[cynyr]

is there a nice easy standard way to find out what codec is inside of the mkv? I'm trying to serve them up over DLNA and the video/audio need to be in a specific codec. Granted I can produce incompatible MP4s still, but I still havn't found an automated tool to open a media file, and spit out what is in it. I need to know which level/frame size/bitrate/etc of every h264 video stream.

Yes I know mediainfo exists, but I have yet to find a machine output option for it. I have little interest in writing a parser for the output.

Re:No reason to change from H.264

[Ltap]

There are few really good cross-platform tools in that regard. I would recommend looking at something like mplayer and finding what library they use to examine containers, then see if you can use that (if you are on a *nix platform, anyway). On Windows it might be substantially more difficult.

Re:No reason to change from H.264

[Snaller]

"Plenty of thin set-top box clients play mkvs already. "

Yeah, but no PC editor support mkvs, only mp4 (and no, mkvmerge being able to split an mkv is not editing)

Re:No reason to change from H.264

[cynyr]

I tend to be trying to use it from python or bash. Mplayer, ffmpeg, and mediainfo all work, but none seems to have a stable machine output option. Heck ffmpeg recently changed its position in the video during an encode from seconds to HH:MM:SS and broke a script I wrote that is using that to determine real world time left of an encode.

Re:No reason to change from H.264

[Osty]

Yeah, but no PC editor support mkvs, only mp4 (and no, mkvmerge being able to split an mkv is not editing)

Huh? That doesn't seem right. But even if that were the case, the beauty of MKV is that it is so easily muxable. Pull it apart, edit the component streams in whatever editor you like (any good editor should be able to handle individual component streams), put it back together. Tada! Edited MKV.

MKV is a container, and a relatively simple-to-mux one at that. The fault is not with MKV, but with video editors that spend time doing the hard work (editing H264 is not easy) and then skimping on the easy part.

Why?

[bonch]

Should I start looking at a rack system?

Well, how should we know what price you're willing to spend? If you're just wanting to know what rack systems are available that can support 10 hard drives and how much they cost, can't you just Google that?

Also, what about location — I could run some cat6 to the garage and move it out of the house, in case noise is an issue.

How could we answer this? We don't know what location is most convenient for you. We don't know how much physical space you have available in your house or your garage. Are your family's file-serving needs so extreme that you can't even rely on wireless networking?

Finally, what about file format, file system, and OS/software? I'm currently running with ext3 and Debian Squeeze. Happy with my audio encoding choice, but not sure about x264 and mkv. I'd also consider different media server software, too. Any comments appreciated.

You're running out of disk space, yet you're encoding audio with a technically inferior format that uses more space for less quality than other formats.

Honestly, based on the rattling off of technical specs, it sounds like this project is more about tinkering and tweaking. There are plenty of pre-built Linux-based media server projects (I'd suggest other operating systems, but I know you'll only accept Linux), but you're not going to accept those solutions because you want to tweak and do it all yourself. For crying out loud, you're running your own family mail server and 8TB file server in an era of Facebook-based communication, web-based email, and cloud storage/streaming services.

Re:Why?

[Anonymous Coward]

A little harsh. The question wasn't posed perfectly, but I'd say I got the idea of what he's looking for: great, contemporary ideas that he hasn't heard of in the home server space. I think he realizes that a rack system is going to be expensive, but that someone out there may be doing it on a tight budget and may have some advice, and since it's going to be long term and multi-user, the cost might be justified.

Personally, I can't help much. I run a cheap retail headless box with 4 drive slots on a shitty uplink. It sounds like this guy wants a permanent solution and doesn't mind getting his hands dirty. I can appreciate that, and hope he finds some constructive advice here, but I'll agree with you on this: we need more information about his actual needs, as his questions don't allow for much outside the box thinking.

My only comment would be on the additional pci comment. If you're going to stuff the gear in the garage, who gives a shit how it's all connected?

Re:Why?

[EvanED]

I do think that this is one of those very vague, under-spec'ed questions. But...

You're running out of disk space, yet you're encoding audio with a technically inferior format that uses more space for less quality than other formats.

If he's running out of space on 8 TB, I'd say that audio is almost certainly the least of his problem. I don't exactly have a huge CD collection, but I do have somewhere around 300 CDs that I have ripped to FLAC, and that comes to under 100 GB. He could have 8 times the number of CDs I do and the audio portion would still only be a reasonably small 10% -- and that's ignoring the difference between Vorbis and FLAC.

Re:Why?

[slick_rick]

@Bonch: Congratulations, you are officially the Grinchy douche on Christmas.

OP: Great questions, I can not wait to see answers. Would love to hear more about what your software solution is for encoding, I've got a bunch of DVDs & Bluerays I would really like to get on the network, but a streamlined rip+encode+publish I have yet to achieve :-/ What are you using on the frontend? I've got various iDevices roaming about, and a Roku that does 720p for the projector, but haven't had much luck with mt-daapd so far.

I do not trust wireless for my projection room, I ran cat 6 everywhere when I moved in as nothing beats the reliability of copper. I've got a basement, so I put the server is right by the patch panel in the furnace room. Noise is definitely an issue. The only thing that scares me about the garage is bugs and what not gumming up the cooling enough to cause catastrophic failure. With a bunch of drives you need a bunch of airflow to keep everything cool, I can just imagine the intake fan sucking up mosquitoes, and coating the CPU cooling block with my dog's blood.

As far as rack vs tower, I started with a rack and ended up with towers. The rack systems just took up way too much floor space, and that space was poorly utilized to boot. It made a very large utility room feel much smaller.

Re:Why?

[InterestingFella]

Rack doesn't make any sense for home environments. It's best used in data-center environments because it saves space and is highly standardized, meaning it also has fast and easy ways to connect input devices and everything else in a environment that hosts thousands of servers. The guy isn't going to be hosting so many servers, so rack doesn't really help with anything. It just costs more and will actually restrict you with size limitations and customization. On top of that you need to get something to mount it to. With a standard box you can just put it anywhere.

Re:Why?

[smi.james.th]

His standard box is already full though. Unless he gets another one next to it. I've never done anything with more than 3 HDDs personally, so I'm afraid I'd be very little help here. Good luck though, it sounds like a fabulous project!

Re:Why?

[symbolset]

You can get a 25-30U rolling rack and put it into the garage. I did. $100 at surplus.

Re:Why?

[mikael]

I'd say it would be better for him to get a separate system for the kids. Some time they are going to want to move away from home and move to college/into their own flat.

At that point, they are going to have to buy their own data storage then and transfer everything across.

Re:Why?

[Anonymous Coward]

The submitter has very specific tastes and would rather keep conversion to a minimum because of the overhead and liabilities involved.

You see, the submitter is storing guro, bestiality, snuff, squirt/scat, and crush videos; all featuring underage animals, so the overhead must be kept to a minimum. He must make do with wired networking only, adding parts to his existing configuration as needed so his horrific pornography may be available to him immediately, whenever he wants to jack off.

I've seen his workstation. He lays a towel over his cheap office chair so he can plant his bare ass on it in comfort. He once remarked that he likes to keep his pants around his ankles so he can pull them up in a hurry when his wife or kids knock on the usually-locked door. The interior of all windows in that room are coated with aluminum foil. On the floor next to his chair lies a long athletic sock crusted stiff by what appears to be dry super glue. A marijuana pipe sits on his desk, next to the razor scratches, with loose nuggets and ampule shells of crystal meth and amyl nitrate respectively. He also keeps a noose handy so his free hand can choke his neck while his other hand chokes his chicken.

Yep, hard drives.

Re:Why?

[pjbgravely]

It is probably because he is using ext3 as a file system. It uses a horrendous amount of space for the journal.

Re:Why?

[sjames]

Agreed. In addition, rack is generally designed with the premise that space is at a premium and that it's well worth cramming as much as possible into the smallest space possible and then compensating for the poor natural airflow with high speed fans everywhere. That's fine if space is that big a premium, but ion a home environment, it's rarely THAT tight. A big roomy tower will run quieter, have less problem with failing fans, and probably will run a bit cooler. It'll also be less of a pain to work on.

Re:Why?

[tysonedwards]

No, it uses 5% of the drive as "reserved" for system-level processes, such as log files, root, temp and other misc data. That amount can be reduced or even disabled completely if the need arises. Other file systems are typically designed with the "that's up to the user" mentality.

Re:Why?

[kyrio]

A full tower can fit many hard drives. My PC-V2100A can fit 12. A mid tower can fit around 6 drives. I really don't see the issue, though. He can just put more 3TB HDD inside of or around his case. He could even build a $200 PC, that's probably 20 times better than his current PC, and throw in a few hundred dollars worth of HDD.

Of course, he's retarded for even considering buying a hard drive at this time. He could wait until the price fixing is over in a few months and get 3TB HDDs for $150, instead of the $400 they are selling them for now.

Re:Why?

[BLKMGK]

Not the submitter but.... i do lots of this :)

Atom machines running Ubuntu and XBMC are my front-ends, I use MCE remotes and HDMI for sound and video. 1080P with an ION chipset accelerated using VDPAU is wonderful. Sound can be a hassle to get working with Alsa but it can be done. If this sounds too expensive or complex or whatever - look at the WDLive that was just revamped. $99, supports local storage, wireless, and network shares. XBMC is way nicer though! ;-)

Backend I mentioned this above - unRAID with many SATA disks, works well and sips power when the drives sleep.

Ripping... Slysoft AnyDVD-HD for decryption of BD (yes Win7). I use eac3to to pull the audio and video from the disks. I use x264 to encode and I wrap it all in MKV containers. When there are forced subs I use BDsup2Sub to get them compatible for MKV. As a front-end for the ripping and encoding I use meGUI. It has a GUI for eac3to and can encode the video and even audio if you want. I have some tutorials on various sites for how I do that, feel free to ask questions though. you will want a fast CPU or lots of time - a movie takes me 2-3 hours to encode. 4cores with hyperthreading running 4+ghz gets me 18FPS or so . You can go faster with lower quality or even use the GPU with some packages but they will not have the same options in my experience. Note that there are some gotcha's. Figuring out which track is the proper video track is a treat and audio can be a hassle too. THD I sample down to AC3 :-( DTS I can handle and so can ALSA. Forget menus, and if you want the director's cut plus others FFMPEG doesn't support MKV's ability to store just the extra scenes and chapter skip - hard to explain but it's how BD stores multiple versions. A raw rip of a BD can be nearly 40gigs or more (forget storage size but SOME will use it). Kung-Fu Panda2 for instance - 17Gig for the raw movie. Animated so it squishes nice - 5gig no sound. AC3 audio so compressed already and it's 450megs for the audio, I leave it alone. I have other movies at 34gigs that shrink to 11 with 800meg soundtracks so you get the idea. A DTS track will be say 1.2Gig, I don't compress these.

For regular DVD I again use Slysoft's product and DVDShrink. I store to ISO format and I do not re-encode. DVD are tiny compared to BD!

Hope that helps some :-)

Re:Why?

[nojayuk]

If you're putting a system into a dirty environment such as a basement then either buy or build an environmental enclosure to put the server in. It's basically a sealed box with large filters for cleaning the airflow through the hardware inside it. Enclosure fans are optional, an overtemp alarm/shutdown system isn't. Replace the filters every six months or so and it should be good.

Re:Why?

[Electricity Likes Me]

Power costs. Having a bunch of always on machines starts to add up. That, plus the progress of technology (by the time the kids move away we might be on to using SSDs for everything) means that it's usually better to try and aggregate costs. Plus it can be an interesting hobby!

Re:Why?

[tgeek]

I have to respectfully disagree here. Racking and stacking my gear made all the sense in the world to me. I have the following (from bottom to top) all in the same approx 22"x30" footprint:

- 2 UPS: 1200W & 1500W freestanding (if one ever dies I might consider replacing with rackmount version depending on costs)
- Epson (blech!) All-In-One printer sitting on rack-mounted shelf
- FreeBSD (not FreeNAS!) NAS server built into 4U Norco RPC4224 24-bay hot swappable chassis. Holds virtually all my spinning disks (only exception being a Windows data disk I use in a gaming desktop). Spare hot-swap trays are only 5 bucks each which makes for easy backup. (Perfect? No. Cheaper than a big ass tape drive and media? Depends. Better than nothing? Infinitely!)
- Dedicated Mythtv backend in 3U iStar chassis (SSD boot, all other storage on NAS)
- 2U VM server
- Keyboard tray
- Monitor on shelf (mythtv motherboard not currently iKVM capable)
- Trendnet 16port VLAN switch
- Shelf for DSL modem and wireless AP

Having everything stacked vertically has saved me a ton of floor space. Could I have collapsed everything into fewer boxes? Sure. But that's not something I cared to do. I like my boxes having their specific functions. Could I have used conventional shelving/storage units (such as bread racks) instead. Yep. But they wouldn't give me the same space efficiency as a 19" rack (might have saved a few bucks though)

Regarding FreeNAS: I have no particular beef with FreeNAS. It's quite good if it does exactly what you want it to do. If it doesn't (for example: if it doesn't mount partitions where you want them to be, or you need some packages not included with it), it's a never ending struggle. Guess it's the age old trade-off of ease of use versus flexibility.

Re:Why?

[jones_supa]

Here's even a nice mini-ITX case that can take 6 drives in it.

Re:Why?

[vlm]

because it saves space

highly standardized

The guy isn't going to be hosting so many servers

Those are the guys three problems.

1) I have the entire "under the basement stairs" space plus two partial wiring closets in my house. Other than hoarding for the sake of hoarding, I can not fill them up. Do not spend money and severely constrain yourself to save space, that you do not need to save.

2) Do not highly standardize. My best purchases have been weird special one off this weekend only deals. Surplus sales, etc. In a corporate environment you Really need to standardize on one specific model of rack hardware to keep the spares situation manageable. This is meaningless at home. Do not turn down the opportunity of cheap GHz and cheap TB because you only collect dell 1U rackmounts or whatever.

3) You're cooling limited at the high end of rackmounts anyway, so who cares if you "could" get a rack sized NAS because it requires licenses and higher voltage three phase power anyway. I could wire multiple 15 amp ckts into my wiring closet, but the space heater effect would cause a fire. Thermal is your primary limit at home, usually. Also you get much more interesting experience with triple redundant physical servers, and better overall system reliability, than with one box three times the size. I have a small compute cluster hosting many LXC images, and by playing games with NFS etc I can back up each container to all three machines and furthermore can redeploy any container to any box based on load. You do need to standardized on all i386 or all amd64 to pull this off with LXC. In summary, get multiple small boxes, not one large box.

Re:Why?

[Hatta]

At that point, they can just VPN back home.

Re:Why?

[keith_nt4]

I don't know, I mean this looks pretty easy...

Re:Why?

[jedidiah]

Yes... "the age of Facebook".

Every one I know that still had a Yahoo mail account got their accounts hacked and then used as a carrier for virus infested spam.

Yes. "the era of Facebook".

Perhaps I don't want Facebook screwing it up.

People put a lot of faith in "the cloud" but the fact is that it really is not really warranted. Big prominent services get hacked. They sell your information. They do stupid annoying things. Then there's the problem that the network itself still sucks. We simply don't have the infastructure yet to "live in the cloud". It's just a marketing pipe dream that clueless idiots are prone to buy into because they like bragging about how ignorant of tech they are.

So. Something you (can) actually control yourself can have considerable advantages.

Re:Why?

[Patch86]

Why buy now, when the march of technology means you're going to be able to buy more and better in 10 years time?

Stick with the one devices, and when your kids need one for their own home, buy them one then. Or give them your old one and buy a new one for yourself. Copying data across from one machine to another isn't exactly tricky (and although 8 TB of data might be time consuming, we're talking days rather than years).

Re:Why?

[cynyr]

A better link to the case the parent is talking about and a link to newegg

here is another one and this one will hold a 5.25 drive as well. I want to like the fractal designs Array case, but I would like at least a slim optical bay.

Re:Why?

[cynyr]

If i put it in the garage I would then need computing gear that could handle from -15F to 110F condensing. You must either heat/cool yours, or live in a very nice climate.

Re:Why?

[pjbgravely]

Thanks, I never knew this. Still I will probably not use ext3 on large data drives.

Arduino.

[Anonymous Coward]

Just kidding.

Re:Arduino.

[Anonymous Coward]

I think you would need to imagine a beowulf of arduino...

sysadmin'ed by natalie portman with grits in her pants who gets hugged by cowboyneal

Re:Arduino.

[TheGratefulNet]

and for a serious take on it, net.search on 'spinmaster DIY'.

I designed and built that to solve some staggered spin-up power supply issues, but it does have a back-end CLI and if you connect your usb-TTLserial cable and run a term program, you can spin up/down any disk you want at the 5/12v molex power point.

add in a database that knows which disk file X is on and you have a nice power-aware system that keeps noise down and only spins up drives (media drives) that are needed for a 'session'.

(the arduino part makes some sense, as its an 'out of band' mgmt interface and can even configure drives before the system has booted; or even via a network if you want.)

Re:Arduino.

[badkarmadayaccount]

Linux auto spins down unused drives.

A little under capacity, but perfect otherwise!

[Zoson]

With drive capacities soaring, I wonder if you'll really need 10 drives.
You might want to try something like a fractal design array. It's a small htpc case for a microatx board. It has mounts for six 3.5" drives, and these days a microatx board will have everything you need, including integrated video for all your playback needs.

File System

[Anonymous Coward]

Well as for file system I would say ext4 since btrfs isn't quiet stable yet. Unless you wanted to go with a *BSD in which case FreeBSD is a solid choice and FreeBSD's implementation of UFS is also rock solid. For a while services like Yahoo and Hotmail where exclusively hosted on FreeBSD servers.

Re:File System

[EvanED]

Another option with FreeBSD is ZFS, which is pretty sexy.

Re:File System

[lister king of smeg]

Depending on how adventurious you are you could go with open solaris if you like zfs.

Re:File System

[PipsqueakOnAP133]

Given that OpenSolaris is already deployed all over the place, it's the least adventurous of the options.
I'm using snv_130 for my personal media server and it works fine for me.
6 drives: 2 on ICH, 4 on sil3124. Pentium Dual-Core Mobile 1.73 on a Mini-ITX board with dual-gigE.

If people are unhappy about the lack of a support community, there'a Illumos/OpenIndiana.
Or just download the free Solaris 11 Express install DVD from Oracle's site for the latest and greatest.

I'm almost tempted to make a 2nd server entirely out of old drives with bad sectors and load it with data from optical media just to see how much life can ZFS get out of drives that should already be recycled for scrap metal.

Re:File System

[lister king of smeg]

Yes but it isn't commonly deployed in home it is usually it is regulated to comersial and enterprise datacenters not home media/file servers but I suppose that is only a mater of scale. So then for fun you could try haiku

Re:File System

[rrohbeck]

btrfs has been very stable for me since 3.1 - with RAID10 at least. Sometimes it feels a little slower than ext4 but overall it works well (4x 3TB in RAID10, 3.1TB in use, 1.3M files right now.)

Re:File System

[Electricity Likes Me]

Last I tried with OpenSolaris it choked trying to get the USB2 drivers up and running (not optional when you don't have PS/2 ports anymore).

I would love to use it (since there's no fast ZFS implementation for Linux yet) but it just doesn't seem practical.

Re:File System

[Anonymous Coward]

This is what I set up at work, for nonarchival storage. FreeBSD 9, ZFS on a 5-disk RAID-Z, samba (with AD integration) and NFS. It's been both fast and solid so far - though I guess the 4-core 8GB workstation I repurposed helps. I've got compression and dedup on, so I expect I should stuff some more memory into it when it starts filling up; so far that hasn't been an issue.

Re:File System

[Massacrifice]

since there's no fast ZFS implementation for Linux yet

Ha Ha! http://zfsonlinux.org/

There you go. Fully native ZFS on Linux. No more FUSE.

Re:File System

[Electricity Likes Me]

Operative word being "fast". Native ZFS on Linux is nice, but it's still not nearly as fast as OpenSolaris's implementation - yet.

Re:File System

[rrohbeck]

The basic premise was always that it doesn't need a checking utility and they're only writing one, reluctantly it seems, because people want one. My system has crashed hard a few times from doing naughty things and the FS never even complained during mount.

raid

[Anonymous Coward]

start playing linux mdadm and run raid 5 array with 1 or 2 spare drives. I built my first 1TB system about 9 years ago with 14 120G drives. ran without major failure for 6 years. Play with the array (remove drives, add new drives, etc) BEFORE you have a drive failure so you understand how to repair the failure. Let me say again, learn BEFORE you have failure. Otherwise, you'll freak out at losing all your data because you screwed something up during repair.

Re:raid

[swalve]

That's what I do, and I'm about to split my "server" into two machines. One, a quasi-SAN with just shitloads of hard drives raided and accessed via iSCSI, and the other will do all the heavy lifting.

Re:raid

[jedidiah]

You should not be dependent on a single array.

You can't just depend on RAID to save your butt. You need to have a backup. Given the size of these arrays, it has to be another array. There's really no other good option. It's kind of painful but unavoidable.

You should never be in the position to panic about your data. Your stuff should be safe even if you need to rebuild an array from scratch.

Re:raid

[Master of Transhuman]

If we're being pedantic, what part of "quasi-SAN" didn't you understand?

And a "shitload of hard drives accessed via iSCSI" IS a SAN. A SAN is nothing but a (special) network card in a PC hooked up via a cable to another (special) network card in another PC with "a shitload of disks" attached to it. If you use iSCSI, you can remove the "special" part...

The fact that the industry likes to make this complicated and expensive is just another example of how screwed up the industry is...

Larger disks

[Anonymous Coward]

Larger disks. 4 TB should be available very soon (maybe now?).
http://www.extremetech.com/computing/108665-hitachi-ships-worlds-first-4tb-hard-drive-sticks-it-to-thor

I use mythtv

[JustNiz]

I use mythtv. It does pretty much everything. I love it.

Re:I use mythtv

[jamesh]

I use mythtv. It does pretty much everything. I love it.

Out of space? Just add another box. The idea that your entire media store is attached to a single server seems a bit old fashioned.

Re:I use mythtv

[Anonymous Coward]

Myth is not suitable for a modern HTPC, because it doesn't support streaming from netflix. That's a fatal flaw for a huge number of people these days.

Re:I use mythtv

[jamesh]

Myth is not suitable for a modern HTPC, because it doesn't support streaming from netflix. That's a fatal flaw for a huge number of people these days.

what's a netflix?

Re:I use mythtv

[symbolset]

Don't most of us get our netflix some other way now? It's embedded in five phones, two tv's, five tablets, six PCs just at my house. They probably think I'm sharing the account by now. Who really needs another way to watch Netflix? BTW: am watching Netflix right now on tv while tapping this out on my tablet.

Re:I use mythtv

[EdIII]

If you really want Netflix cheap you can buy a Western Digital Live TV Plus. They are under $100 and do extremely well with Netflix and a bunch of other apps. I have gone through several firmware updates and they keep adding new apps. In addition to the apps you can do streaming from just about any kind of server and USB connected flash drives.

HDMI and even optical audio out if you needed it. Netflix has a lot of HD titles, plus 5.1 audio on quite a bit too. I believe the Ethernet is gigabit, but I don't know off the top of my head.

Only thing it does not do well at all is DVD ISO files.

For under $100 though with a nifty little remote you will find yourself using it for more than just Netflix on a regular basis.

Consider it a nice little addition to a HTPC.

Re:I use mythtv

[Elrond, Duke of URL]

I am curious about this. I've been a MythTV user for many years now. I don't *need* Netflix, but it certainly would be nice to have. Part of the draw of a well built HTPC is having everything in a single place controlled by a single remote.

But, you're right that MythTV does not natively support Netflix. Maybe it could be accessed via MythBrowser? Or is a real Win/Mac (on a PC) client necessary?

Is there any other Linux alternative to getting Netflix running? Or, for that matter, any other movie streaming service? I believe MythBrowser is built on top of Webkit and that it can handle the Flash plugin. Certainly, that's enough for YouTube (already handled by MythNetbrowser), but maybe not more.

Google tells me that some people have done this by having their Mythbox launch a WinXP virtual machine to run IE to run the Silverlight Netflix player. The author seems to indicate that, overall, it is not hard to implement, but the CPU requirements are fairly high. Hmmm...

Re:I use mythtv

[Bent Mind]

But, you're right that MythTV does not natively support Netflix. Maybe it could be accessed via MythBrowser? Or is a real Win/Mac (on a PC) client necessary?

Netflix depends on Silverlight. Silverlight is not supported (or usable) on Linux. Doesn't even work under Wine. A good alternative to Netflix is Hulu. It uses Flash for video.

Re:I use mythtv

[slackware 3.6]

There is the Nintendo Wii as well.

Re:I use mythtv

[Elrond, Duke of URL]

Hmmm... that is unfortunate.

The suggestion to use Hulu is a good one, but I don't think it will particularly fit my needs. Hulu is focused much more on television shows rather than movies. According to their list, Hulu Plus has only 1800 movies available, of which only ~700 are in HD. That's not enough to justify the monthly cost since there is only a small subset of those movies that I will want to watch.

Re:I use mythtv

[BLKMGK]

Well... you do know MythTV can run in Windows now right? I'm betting with a little work NetFlix could indeed be up and running....

http://www.mythtv.org/wiki/MythTV_on_Windows

Re:I use mythtv

[lister king of smeg]

Android emulator/Sdk is about it if you dot want a windows box or vm (or hackintosh) when it comes to netflix -(not counting embedded)

Re:I use mythtv

[Pentium100]

Putting everything in a single server has its advantages - lower power consumption for one (because you do not need to heat additional CPUs, RAM etc) and less physical space used.

I used the "just add another box" method some years ago, so now I have ~3TB (in >10 hard drives) and 6 computers (including the main one), total power consumption ~1.2kW. I would love to replace 5 of them with a single server, but servers are really expensive, especially the ones that have more than 4 HDD slots and a rackmount (seriously, I have not seen a 3-4U server that had a lot of hard drive slots, but less than 4 CPU sockets, I do not need that much CPU power, but all 1-2 socket servers are 1-2U, which means less space for HDDs). I would probably use that server for ~10 years, so it has to be good.

The advantage of multiple servers is that if one goes down I only lose some functions.

Re:I use mythtv

[Electricity Likes Me]

You should check out the Norco cases. They're 4U rackmount units with 24 hot-swap HDD bays, designed to fit regular ATX motherboards. Very cheap for what you get (also include a ton of fans). The build quality is not spectacular, but for a home system where you're not doing a whole lot of swapping they're great.

Re:I use mythtv

[Wagoo]

You should check out the Norco cases. They're 4U rackmount units with 24 hot-swap HDD bays, designed to fit regular ATX motherboards. Very cheap for what you get (also include a ton of fans). The build quality is not spectacular, but for a home system where you're not doing a whole lot of swapping they're great.

^^ This. Having hotswap also makes a big difference when you start having lots of drives. I'm still using a Norco RPC-4020 from about 3 years ago, 20x hotswap bays.. with the backplane mounted (8cm) fans running at 5V. Makes it acceptably quiet.. a lot of the server grade hardware will be set to tornado mode by default, as who cares about noise in a datacentre.

Re:I use mythtv

[RanceJustice]

So the Wii and Android versions of Netflix's client run on Silverlight? That can't be right.... since Netflix "instant" is already on many devices, maybe the time has come for the Linux and perhaps specifically the Myth community to seriously engage with Netflix on getting a version that's viable for Linux? After all, if they can make it work on Android its not too big a jump. I was given a new Samsung HDTV and it even has Netflix "built in" if one connects the ethernet jack on the TV to the network; I was to understand these embedded TV systems were mostly Linux-based. The 5 year ols Philips HDTV I'm using as a monitor is forthcoming about it - the manual and about/options screen on the TV itself has Linux kernel information and GPL stuff. The whole "it can't be financially viable" or "we don't have the capital to invest" arguments have gone out the window after all this time and development on other platforms; I don't think it is unreasonable for a full-featured Linux client.

Re:I use mythtv

[Pentium100]

My room reaches 34-40 degrees in the summer, I usually replace the fans of my computers with the fastest ones of that size, so tornado mode is OK (I have two servers and they automatically reduce the fan speed when it gets colder (say, in winter, when I can keep my room at a pleasant (to me) 15 degrees).

The case looks interesting, but no local computer store has it and buying it on ebay would cost a lot in shipping charges.

I also like the additional features of server-grade stuff - redundant power supplies (which means I can clean the dust from them without turning off the server) redundant fans and so on.

My problem is that new servers are too expensive (and a bit overpowered). While older (not 10 year old like the one I ma using now, but, say, 3-4 year old) would be OK in price and CPU power, they only accept the expensive SCSI hard drives, while newer ones can use cheaper SATA drives. So, if I wait a couple of years, then the current servers will become the "older" and cheaper ones, but still with SATA support. Then I will buy one and fill it with Seagate Constellation ES or WD RE drives.

Re:I use mythtv

[jedidiah]

> Myth is not suitable for a modern HTPC, because it doesn't support streaming from netflix.

Then buy a $60 appliance that's cheaper than most "trendy remotes".

Gear is cheap these days. Why settle when you can have both? Buy the appliance for what it's good for and don't try to turn it into something it's not.

Re:I use mythtv

[rwa2]

The Roku box is the only Linux device I've heard of that can stream Netflix. The Android-based devices seem to work OK too.

I tend to not really care about all things TV&Movies, but if I really needed to have netflix working under Linux, I'd probably just run Windows under Virtualbox or something. The video bandwidth probably isn't high enough for HD, but oh well.

I would simply go find my entertainments elsewhere, but I'm one of those weird people who find entertainment in searching for entertainment.

Re:I use mythtv

[White Flame]

Seeing as you're just piling boxes around, why not just pile harddrives around the computer? As long as your motherboard has enough SATA ports, you don't *need* to have a fancy case with physical slots for all the HDs. Consumer-grade motherboards with 6-8 SATA slots with a few more eSATA are not uncommon.

Then again, if you only have 3TB, buying some cheaper 1-2TB drives, integrating them into your mix and eventually moving your data over, should allow you to power down a lot of your equipment especially if you're mostly using it for storage, not processing.

Re:I use mythtv

[jamesh]

Putting everything in a single server has its advantages - lower power consumption for one (because you do not need to heat additional CPUs, RAM etc) and less physical space used.

Do you think the OP really needs to keep his/her 8TB of disk powered up and online _all_ the time, just on the off chance they might want to watch something right now that can't wait 30 seconds for a second box to boot up?

We're still using DVD's at our house and the kids seem to watch a new movie a few times (or a few hundred times... :) in the first month or so then, with a few exceptions, not really watch it again for ages. All those older movies could go on a box that only boots up when required. In fact in our house, the primary server could probably get away with a single TB of disk and still have room to spare. Now _that_ would save power.

What puzzles me though is how you'd ever have time to watch 8TB of media. Assuming it was DVD quality (which it's probably not), we're talking about 6 months of non-stop watching, and then keeping the media just in case you might want to watch it again. How many movies have you ever seen than you'd want to watch again? And then again after that? For me it would be less than half, probably much less.

I suspect that someone with 8TB of online media storage is a bit of a hoarder ;)

Re:I use mythtv

[Pentium100]

I use LTO-2 tapes to archive the stuff that I already watched (as they take up less space than DVDs).

How many movies have you ever seen than you'd want to watch again?

Movies? Probably not that many, I do not watch that many movies anyway, mainly TV shows. TV shows? Sure, I watched B5 and DS9 like 6 times each.

As for the archiving, I archive because I do not know if I am able to download the same movie in the future. Maybe everybody stops seeding (the downside of BitTorrent compared to, say, Gnutella or ed2k, is that you have to specifically seed the file, instead of pointing the program to your folder of things you want to share and letting others download them), maybe the internet gets locked down, I cannot predict the future, but sure as hell, I am not deleting those Law & Order episodes that took over a month to download using eMule.

Also, HD video takes up a lot of space. A 1080p (x264) movie is 10GB or more, a BD rip is ~40GB.

I also have a record, cassette and reel tape collection, as well as a lot of music TV shows and concerts recorded from TV on VHS (I still record to VHS now). Add to that my LTO tape, hard drive and DVD collection and you will have my "external memory".

Re:I use mythtv

[jamesh]

As for the archiving, I archive because I do not know if I am able to download the same movie in the future.

Ah. I made the flawed assumption that people would actually own the shows/movies they were watching and could just re-load from the original DVD.

Re:I use mythtv

[JustNiz]

why would you assume because I use mythtv I have everything in one box? huh?
I have part of my mythtv stuff on a NAS.
Like I said, Mythtv rocks.

Re:I use mythtv

[jamesh]

why would you assume because I use mythtv I have everything in one box? huh?
I have part of my mythtv stuff on a NAS.
Like I said, Mythtv rocks.

Yeah I probably shouldn't post late at night. I meant that as an advantage of MythTV as opposed to the OP who has run out of space on his media centre and doesn't know what to do.

Re:I use mythtv

[Pentium100]

Really? 8TB of hard drive space just because the guy is too lazy to go grab a DVD when he wants to watch it? Unless he keeps his DVDs in another house, I doubt it.
Then again, the legit DVDs that I have, do not start playing the movie, but play a bunch of ads instead, so I could see the point of ripping the DVD so that the ads could be removed.

Re:I use mythtv

[JustNiz]

Dude what?
Do you realise that you can get 4TB on a single relatively cheap drive these days?

I RECENTLY bought a 4 bay NAS (cost me about $250) and 4 3Tb SATA drives when they were about $120 each. so for about $750 I have a 12 TB network raid that consumes about 50W when its doing heavy I/O. Less on standby because the NAS box supports auto spindown.

Re:I use mythtv

[matthewd.net]

Really? 8TB of hard drive space just because the guy is too lazy to go grab a DVD when he wants to watch it?

I'm currently contemplating exactly that. But I'll admit, I am pretty lazy. (And technically, I think I only need about 6 TB at the moment.)

Avoiding the "because you bought this, we'll lecture you on why you should've bought it" intros does have some real appeal too.

Separate servers

[obarthelemy]

Instead of tacking function after function onto the same server, I'd encourage you to use several small ones, including the $35 Raspberry Pi. That way one one piece of software starts to go haywire, it doesn't bring the whole shebang down with it.
Also, I'd go for fewer, larger disks. As long as you do backups it's not more risky, and it's a lot more practical. HDs reach 4TB these days, so you're talking 2 HD for your existing data, a 3rd one more more capacity, and double that for backups.

Re:Separate servers

[BLKMGK]

Sure! You just have to get the right hardware and after tinkering with ESX for awhile I gave up on it. None of my onboard NICs were supported! No thanks, wake me when it's running on broader hardware support.

Re:Separate servers

[jedidiah]

What's so complex about having your mail server sit in a different box? Really?

As far as the media stuff goes: complexity is mostly dependent on the sophistication of the playback devices. The more stupid they are, the more of a bother it will be.

Figure out what really needs or benefits from being "unified" and let the rest be split up if that helps.

you may want to have raid 5 or 6

[Joe_Dragon]

10 disks in a raid 0 type setup is a big risk. Also lot's of pci cards eat up PCI bus io and maybe even the same io used for the network also your board like only has 100M e-net. Now if your system has a pci-e slot then a 10 port non raid card + software raid may work and is cheap then a raid card. But you may also want to get a newer MB + cpu most new amd and intel boards max out at 8 sata ports. 8 ports may work out ok or you can get a new board with a dual core cpu + a raid or non raid card also a new MB will get you GIG-E + pci-e IO.

Re:you may want to have raid 5 or 6

[sirsnork]

Yup, start again.

Pick a board with plenty of SATA ports, put a modest amount of RAM and CPU in it. Make sure it's got PCI-E slots (what hasn't these days) and go from there.

Use bigger drives than you are currently, it's a bad time to buy drives so wait if you can but just build a new box from scratch and save yourself the headache of trying to migrate drives or retain data while upgrading drives one at a time in an existing array.

New machine, 3TB drives x as many as you want (6 would about double your capacity), add a 4 port PCI-E SATA card if you need it and rsync all the data across, job done

Re:you may want to have raid 5 or 6

[93 Escort Wagon]

Raid seem silly in a home setup. People see it as a backup solution when it's not, and I doubt a few extra tenths of a percent of uptime is really significant at home. Just have one or two big disks for all your media and stuff, then buy the equivalent to use for regular rsync backups. You can get a third to take "off site" occasionally (e.g. your sister's house, a friends house, a drawer at work) if you want to be really careful - just rotate the backup disks every so often.

Re:you may want to have raid 5 or 6

[swalve]

I like it for expediency (no need to restore when a drive fails) and for simplicity. I found myself with single drives in a bunch of machines, and my free space management got to be a huge waste of time. It became easier to just concatenate all my storage into one volume with one "xxxGB free space" indicator and when that starts to fill up, I start replacing hard drives.

Re:you may want to have raid 5 or 6

[nabsltd]

Raid seem silly in a home setup.

Once you start to use more than one hard drive worth of space, RAID is pretty much required unless you really like to spend days restoring data when a hard drive fails.

Re:you may want to have raid 5 or 6

[rrohbeck]

Failure rate of disk drives in consumer environments is about 3%/year. With N drives you'll see N*3% failures per year. You do the math.

Re:you may want to have raid 5 or 6

[lister king of smeg]

That's right raid is silly for home heck so is back up drives just upload everything to the pirate bay and you will have it backed up to thousands of other drives where it can be downloaded at your convince at at a later date. Torrents the redundancy of raid and the security of off site back ups all in one easy to use format.

Re:you may want to have raid 5 or 6

[CSMoran]

Not N*3%. Learn some basic maths. It will be 1 - (1 - 3%)^N.

Are you implying I can never have more than 1 failure per year, no matter how many drives I have? Cool!

Re:you may want to have raid 5 or 6

[jedidiah]

Are you kidding? That kind of crap would be even less tolerated in a business environment. Never mind a home environment. Companies use RAID specifically to AVOID the kind of bullshit you are describing.

Clearly you haven't used it in either sort of environment.

For media server use, you don't need a lot of performance or a lot of redundancy. You can cut a lot of corners because you aren't trying to replicate someone's TPC benchmarks.

One of the benefits of "cheap" is that you can easily afford TWO.

Re:you may want to have raid 5 or 6

[rwa2]

Yes, RAIDs do not eliminate the need for backups, but there are plenty of benefits to using it on a home Linux server. Here's my typical setup using Linux SW raid with 2-4 disks partitioned somewhat like the following:

/ on RAID10. (The Linux RAID10 with the -f2 option). Allows reads in parallel, lowers latency under load (the driver will give the request to the head that needs to move the shortest distance, or the disk that is not busy serving some other request), and still allows your system to boot when (not if) one of your drives fails. /var/tmp on RAID0 . I don't always set this up, since it's easier to just use tmpfs for stuff like /tmp nowadays. But it's useful to have some scratch space on your system for compiles or things that write a lot of temporary things that get thrown away. /home on RAID5 (or 6 if you've got lots of disks), so the remaining bunch of data available on the drives gets pooled together and has decent performance. Also usually make /usr/local a symlink to /home/local , which tends to make it easier for me to carry my stuff between distros.

Had my system running this way for over 10 years now and it carried me through 2 disk failures. Mdadm is your friend.

Re:you may want to have raid 5 or 6

[cynyr]

or run software raid and skip the raid card, put the system in a back room/basement and who cares if all the drives spin up when I watch a movie?

As a note my current setup is just that, a 3 drive software raid 5 in a micro-atx case in the back room until the onboard nic died. I'm waiting on the replacement motherboard but at least with software raid if the sata controller dies it's a simple swap it out and move on with life as mdadm doesn't care which controller or port the drives are hooked to.

FreeNAS

[Anonymous Coward]

For what it's worth, I've had a FreeNAS box running for about a year now and I've loved it. I recently switched from FreeNAS .7 to 8.0.2 which does remove a lot of features, so I added a Windows based server to supplement that until FreeNAS 8.1 or 8.2 comes out, I'm sure you could have a similar setup with Debian instead of Windows. My main reason for FreeNAS has always been ZFS with a decent RAID-Z setup, which I was very glad for when one drive failed a couple of months back. If all you really need is more storage then perhaps a FreeNAS based supplement would work well, link aggregation and iSCSI are pretty neat things.

HP Microserver

[quenda]

For a slightly more sane solution than rackmounting at home, consider the HP microserver.
Very low power (12W CPU), small, quiet, cheap, server grade, no Windows tax, holds four pluggable 3.5" drives plus optical (which some people swap for a 5th HDD for RAID5.)

http://blog.thestateofme.com/2011/05/14/review-hp-microserver/
http://www.silentpcreview.com/HP_Proliant_MicroServer
http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-4237916-4237918-4237917-4248009.html
http://forums.overclockers.com.au/showthread.php?t=905262

If 8TB is full, you need to stop the obsessive collection of warez/pr0n/torrentz you are never likely to watch again.

Re:HP Microserver

[Anonymous Coward]

"8 TB should be plenty for anyone."

Re:HP Microserver

[Anonymous Coward]

I've got one of those. Keep in mind that they consume about 55W of electricity while running, and do not support suspend-to-ram (ACPI S3). That means it will cost you about $5 per month to run it, unless you actually shut it down or hibernate it (ACPI S4 is supported). Fortunately, 55W is much lower than most other computers suitable for your purpose.

Re:HP Microserver

[EdIII]

If 8TB is full, you need to stop the obsessive collection of warez/pr0n/torrentz you are never likely to watch again.

As opposed to the obsessive collecting of physical media that can be scratched and takes up 10x+ the space?

My family since the 80s has amassed literally 10k CD/DVDs as well as almost 100 laserdisc titles. Not to mention a buttload of VHS tapes that we offloaded years ago.

It has all been converted to digital storage. Since it is on multiple RAID 5 devices and I run a cron job that checks the MD5 sigs against a database I know that it is in good condition.

Of course this requires constant rollover of the data from hard drive to hard drive. Half the drives have failed over the years and it has moved between multiple NAS systems. We still have all the data.

In addition to that, we have over 100k family photos collected from all of our relatives scanned and tagged as well.

Our collection is nearing 20 TB. With the low cost of drives we have backups in lead lined containers in safety deposit boxes at two banks. We swap them out every year or so adding to it. I am really looking forward to long term archival storage that is write once and designed to last 100 years plus. I'll pay for that.

Now I know you may be thinking obsession, but we *paid* for it. Paying twice for music or movies is just plain insane and we never fell for the HD/Bluesuck shit they were shoving down our throats. Well my parents did, but Spiderman solved that problem the first time it could not be played because the encryption changed. Since then they are back on DVD only and we are waiting for a HD storage method that does not involve constant Big Brother monitoring and DRM in our houses.

Then there is the most obvious benefit of all. You only have to rip the music or movie one time. Been years since we bought an actual CD, but you get my point.

The convenience of having all of your media at your fingertips without touching physical media is pretty damn nice.

Guess how much storage space you need for thousands of DVD/CDs when they are packed into spindles and put into storage? A heck of lot less than you would expect. Fits in a closet.

Re:HP Microserver

[Anonymous Coward]

Maybe you should work at a museum, clearly your archival talents are being wasted at home.

Re:HP Microserver

[EdIII]

Perhaps so. There really is no choice unless you want to eventually become a victim of Big Content. The GP strongly implied piracy as the only cause of such massive storage requirements. It's not at all.

It has to be stored *somewhere* and since managing physical media is a pain in the ass and *every* single use causes damage, another solution must be found. Be it ever so slight, there is damage with physical media. It makes no sense if we own the legal entitlements to a copyrighted work to keep it stored in a manner that constantly degrades. We have the right, both legally and ethically to take measures to protect our legal entitlement those copyrighted works. Now it would be nice if copyright law was more sane and works actually made it into the public domain within a reasonable time period. My family would pay $50 a month for access to a massive online public library of books, music, and movies. That would be more cost effective than creating it ourselves.

What about our family heritage? It's not just photos. Artwork, birth certificates, personal documents, etc. I can see within the next 20 years that diaries would be easier to keep, and more private than FaceStupid, with software designed to do it.

The article brings up a good question. What is the best way to keep a massive digital library in good condition and the most convenient way to access the data?

I gave my advice for how my family handles it and I really wish people would just drop Blueray support all together. Never buy into a copyright model where your legal entitlements and ability to protect said legal entitlements is limited by greed and unethical conduct.

Re:HP Microserver

[BLKMGK]

Umm, why not strip the DRM from the BD and store those too? the picture really is much better even if you re-encode them for smaller storage. I have a post above about this but yeah it's doable. I also have about the same amount of storgae as you but it's not backed up other than the unRAID software's protections. It's simply too expensive to have a full dupe for me :(

Re:HP Microserver

[BLKMGK]

You DO realize that normal SD DVD had DRM too right? It had the SAME sorts of restrictions that BD has now and so did HD-DVD! The only difference was that the DRM was easier to defeat because they couldn't swap keys as they were compromised. You're not doing anything different now with DVD than you would be with BD. Strip the DRM from BD, re-encode if you want it smaller, and then watch that much nicer picture. It's harder, it's more time consuming, and you don't end up with menus and advertisements. You can keep the director soundtracks though if you want :-)

I'd also argue that a laser beam does no damage to a DVD or BD. The damage these gets is from oxygen leaking into the media and oxidizing the metal. This is real, it's happened to some of my CD. Scratches happen but so long as they aren't on the label side they can be recovered with a buffer - a car wax like Nu-Finish with distillates works wonders too.

BTW - where did the GP imply piracy at ALL in his posting? I didn't get that and if you saw MKV and thought that meant piracy then you need to get out more. MKV is THE best container to store video content in. It supports most codec, supports subtitles - both forced and normal, supports multiple videos if you want, can support multiple soundtracks, and can even support menus if people would make tools for it. I see nothing in his post about piracy, I DO see someone with the same issues I have archiving BD which is apparently a medium you've not educated yourself on enough.

Re:HP Microserver

[EdIII]

BD is never an option for multiple reasons.

1) Even if doable, which keeps changing, a true BD backup takes up a *lot* more storage. I have seen torrents that are 25-30GB for a BD rip. Why even download it or rip it?
2) To rip it, I would have to purchase it. I would never do so because that would be voting with my wallet in the wrong direction.
3) Re-encoding has variable quality depending on the algorithm and the parameters. Piracy groups do it better, so I would never do it myself. In the end I would get something marginally better than DVD quality at 720p.

It's about not supporting their business model at any costs. I don't purchase Blueray technology at any level from player to content. It is getting worse too. They are pushing towards an Internet connected model so they can keep the encryption algorithms and keys variable with an undesirable side effect of direct control in my house and an unacceptable invasion of my privacy. I don't need their fucking permission or their knowledge when I decide to watch a movie . I purchased that legal entitlement, preferably with non-traceable cash, at the store.

Once I leave the store with the physical media, my legal entitlements are permanent and will only cease when the work enters the public domain. Anything else is unacceptable and they can blow me if they want to charge per use, per media shift, per time shift, etc.

When you pirate or purchase something based on Blueray you are endorsing, encouraging, and enabling a business model based on unethical ideology on the part of Big Content that has the ultimate goal of destroying the Public Domain and changing the philosophy of copyright to be permanent ownership and absolute control of content regardless of any and all considerations.

Of course you can point out that DVD has similar caveats, but DVD is known and considered by all sides to be so broken that the encryption and PUOs are laughable little speed bumps and not a serious impediment to any action by the consumer, infringing or not.

Re:HP Microserver

[EdIII]

You DO realize that normal SD DVD had DRM too right? It had the SAME sorts of restrictions that BD has now and so did HD-DVD! The only difference was that the DRM was easier to defeat because they couldn't swap keys as they were compromised

I covered that in my other response to you.

I'd also argue that a laser beam does no damage to a DVD or BD. The damage these gets is from oxygen leaking into the media and oxidizing the metal. This is real, it's happened to some of my CD. Scratches happen but so long as they aren't on the label side they can be recovered with a buffer - a car wax like Nu-Finish with distillates works wonders too.

I never said laser beams did damage. The act of putting the media in the player does damage, even if ever so slightly. Your point about oxidation is also understood. Which is why even the best storage methods, short of a vacuum, will only marginally prolong storage life. Ultimately, shifting and validating the data across different mediums is the only long term storage solution.

BTW - where did the GP imply piracy at ALL in his posting?

I thought the reference to warez and torrents was more than enough. Torrents can be non infringing, but warez is kind of specifically stating infringement.

I DO see someone with the same issues I have archiving BD which is apparently a medium you've not educated yourself on enough.

I am very well educated on BD. That is why I choose to boycott it at all costs.

Re:HP Microserver

[Any Web Loco]

Hi - I have one of these and am looking tower up similarly so am interested in what you did but your link doesn't take to direct post - can you amend?

Re:HP Microserver

[fingon]

I really recommend ZFS for that - RAID5 is inferior to RAIDZ, not to even mention RAIDZ2 (allows for blowing up two disks). Also has built-in scrubbing functionality and metric ton of other nice features.

If I was storing that amount of data, I'd just throw say, ~16 2TB drives at it, make 2 RAIDZ2 pools out of it (thus, 6 out of 8 drives in each actually used for storage, two for parity ), and have 24TB of capacity, out of which 2 drives could blow/get corrupted and still no problems, and in best case it would even survive explosion of 4 drives ;)

Re:HP Microserver

[jedidiah]

It's also about handling it. Any time you touch the media, you run the risk of damaging it. It could get scratched coming out of or going into it's original packaging or the media player. Drive trays like to retract while a disk is half way out. Stuff even manages to get damaged in transit to the store or your house.

Although mainly it's about convenience.

iTunes or Netflix convenience without a lot of the limitations or high prices. 5 minutes spent ripping something is not such a bad deal if it's unavailable at iTunes or is a much better bargain at Amazon or Target.

Plus there's the whole "multi-room" thing. Conventional DVD/BD players don't keep track of what episode you're on or where you left off in the last one and can't start you up again from where you left off in another room.

Then there's the whole "one interface" thing. You can do that even with a Linux based HTPC setup that you likely will never have with a collection of randomly acquired appliances.

Re:HP Microserver

[BLKMGK]

Ah, so you're willing to support them with DVD because it's easier to break but not BD because it's harder?

As for storage size, yeah HD video takes more disk space. It takes more disk space because there's MORE data. If you're happy with SD quality then fine ignore BD but don't bitch when others want that quality. Frankly SD looks like crap on a big HD display which is what many of us, myself included, have moved to. Encoding to reduce size by a third or more is EASY albeit time consuming. It doesn't look like 720P either unless you want it to. Some BD movies, especially animation, squash down to almost the same size as DVD and the quality is much much better than DVD, H.264 is a way better algorithm and it shows. the quality is as good as you make it, shrugging and saying that pirates somehow do it better is laughable, they use the same tools as you would and I do. As for the size of BD rips, I wouldn't know what a download size would be because I've not ever downloaded one. If it's 25-30Gig then it's the entire movie in it's native format, too big and filled with extras I'm not likely to want. I'll make my own from original media thanks...

So yeah, I AM saying that by purchasing DVD you're supporting them in the same way - you are! Sure they would love to make BD somehow an internet connected model, when that speedbump occurs I'll happily cease buying and so will most everyone else - they will hit a brick wall. In fact they have slammed into a wall a few times now with BD and had to back off. As it stands right now I no more need anyone's permission to watch BD as I do DVD, when they try to force that it won't last long...

Re:HP Microserver

[BLKMGK]

ah so not the original Slashdot post but some post here referring to warez, yeah fine. I'd agree that shifting to a different medium is the way to go, works for me - covered that too. You may think you're educated on BD but you sure don't seem educated on encoding if you somehow think a pirate somewhere does it better...

Re:HP Microserver

[FauxReal]

What do you do with the physical media? Store it in the attic or something? Maybe stack it up into the shape of a couch and throw a sheet over it?

Re:HP Microserver

[EdIII]

Ah, so you're willing to support them with DVD because it's easier to break but not BD because it's harder?

I am willing to support it because it is fundamentally broken and no longer DRM in any meaningful sense. Everybody knows this, especially Big Content. When I purchase a DVD they know they have no control, and I know it as well.

When I purchase BD, they see it as support, or complacency, and it encourages them to continue. Purchasing a lower quality alternative to bypass more effective and restrictive DRM does send a message.

If you're happy with SD quality then fine ignore BD but don't bitch when others want that quality

I bitch when other people take the path of least resistance and keep buying into DRM models that only continue to hurt us. Just because it is increased quality is no excuse for being part of the problem. If nobody purchased BD at all, then they might start to change Big Content's mind.

shrugging and saying that pirates somehow do it better is laughable, they use the same tools as you would and I do.

It's laughable that you think it is laughable. That's like saying I can walk into a metal shop and use the tools the same way the worker does. Nooooo, more likely I would walk out missing some fingers.

The piracy groups are professionals at it. I should know having spent over 50 hours with testing on a headless CentOS server getting the parameters exactly right for an encoding process.

Knowledge and experience make all the difference and if you are so absorbed with quality you should be able to tell that a BD release was rushed because of the artifacts and "waterfalls" in the movie. That was the studio not spending the time and money for a quality encoding job.

As it stands right now I no more need anyone's permission to watch BD as I do DVD

Yes you do. Right now it is not Internet connected, but there is a problem in that not all BD players have the keys to all BD movies. Having to send out for a firmware DVD, or burning your own is not the same as permission, but is just like groveling and begging to me.

If I have to firmware update my player a couple times a year because they are battling piracy, I say fuck it. I'll stick with DVD where there is no battle.

Re:HP Microserver

[EdIII]

I said it in the other post, but I don't understand why you think a pirate would not be better at it?

They come across nearly every BD release and have to re-encode that content. That is a constantly developing skill that keeps them up to date on the latest decryption, codecs, tools, etc.

The pirate is automatically going to be in the top 1% of all people capable of doing the job. You may have a good understanding, but how much time did you spend to get it? Do you work with codecs and encoding on a regular basis?

In addition, piracy groups release the movie at different quality levels. You have the rip that is not much larger than a DVD rip, 1-2.5 gigs. You also have rips that are around the DVDR size 4-6 gigs, and then you have the full ISO levels that are the 25-30 gigs.

They are using the source in multiple ways and that does require a more sophisticated understanding of the encoders and their parameters.

Having played around with various encoders in Linux there are large MAN pages and websites dedicated to just explaining those parameters and how it affects encoding. You must know that, and you certainly imply that you do .

So how would a pirate not have a higher level of sophistication on average and not be very likely to be considered a professional?

Re:HP Microserver

[EdIII]

Wooden poles. Microfiber on the bottom and top. You can stack up over a thousand in just six feet. Very easy to build. In a regular closet you could put half dozen of them in there real easily.

Get a strong wooden base about 2 inches thick. 5 six foot wooden poles about 15mm. Easiest to get 1/2 inch. There is some play, but that is where the 4 poles come in. You put those in a square designed to give about 1mm of tolerance around the CD/DVDs. Not all are exactly the same size and temperature allows a slight variation (really slight). Attach it to a top wooden "cap" with tapped holes in the base. You can even put cross supports every 1 ft if you wanted with wooden rods tapped into the poles. I recommend that for stability.

Once it is filled you can tighten the whole assembly down and it is really stable to move if you need to as long you keep it straight up. I don't recommend moving it to much though. 1/2 inch poles are not that strong considering the weight. If you want to go all out you could use metal, but I think that is overboard.

Lately I have been debating about a strong plexiglass enclosure that I could create a vacuum in to prevent slow oxidation damage.

My parents have 7 of them in a closet with an 8th one being filled. Each one can store 1500 CD/DVDs. It's going to be quite some time before the 8th one is filled. We only amassed that many because of music over nearly 30 years. Movies accumulate much more slowly.

The pictures and heirlooms take up quite a bit more storage than CD/DVDs. Those we do have in the vacuum seal bags to prevent oxidation.

Of course a single fire takes it all out. At least we will have digital backups offsite. I maintain an entire copy myself.

It would be nice to have a fireproof safe, but there is not anything that large on market that does not cost as much as a freaking small car.

Some say the world is going to end this next year, but what if it doesn't? My parents and I like the idea that with technology getting better all the time we can keep our entire family history, as it is growing, for generations without degradation and multiple synced copies.

The CD/DVD storage spindles are just one part of it.

Re:HP Microserver

[BLKMGK]

I'll say this one more time - no I do NOT need anyone's permission to watch BD. My players have never been updated, I own NO players. I rip the silly thing, encode it, and I watch at my leisure. The decrypting software, which has zippo industry affiliation, is updated occasionally.

Lastly, the encoding process isn't that hard. X.264 is well documented, terrific profiles exist, and if you look around you can even figure out what it is that "pros" are using for options if you really wanted. 50 hours sounds great but if I added up the time spent on encoding my stuff you'd fall over. Once you have the process it's just rinse and repeat and I've been doing this since HD-DVD first came out

Re:HP Microserver

[cpghost]

If I have to firmware update my player a couple times a year because they are battling piracy, I say fuck it. I'll stick with DVD where there is no battle.

Well, with DVD, there's battle too. I've had to update the firmware of quite a lot of DVD players so they become region code agnostic...

Re:HP Microserver

[pyite]

Since it is on multiple RAID 5 devices and I run a cron job that checks the MD5 sigs against a database I know that it is in good condition.

I applaud your desire to maintain integrity, but if something goes bad, you have to manually restore it. I, too, keep MD5 or SHA hashes of important stuff, but I rely on ZFS and 'scrubbing' for verifying integrity. The benefit is if a bit does flip, it will be corrected, and I will be told.

You should consider it.

Re:Mac mini or apple Tv

[Kagetsuki]

I'm sorry, did you read the summary? Do you honestly think a Mac mini is a step up from that or solves the problems presented?

Check out UnRAID

[Anonymous Coward]

Check out UnRAID here. It is open source and lets you add disks more easily while providing redundancy. You should also check out the HTPC forum on AVS Forums.

Re:Check out UnRAID

[BLKMGK]

unRAID is NOT OpenSourced. It does run on Linux but it has closed source portions and it is locked to a USB key that you register. If the key dies Tom is damned good about replacing the license for you - been there done that. Tom does comply with the GPL, any code he modifies that's GPL is included with unRAID. His closed source stuff he keeps to himself and for the service he provides I'm okay with that. I run two of his servers and he's been very helpful over the many years. Slow to update software, bug fixes for new hardware can be slow too, but I have no complaints....

My setup

[inkrypted]

I have been using Ubuntu server and Playstation Media Server (Don't let the name fool you. As for storage use XFS because it's tailor made to server large files. For storage these racks have colling fans and are allummium http://www.newegg.com/Product/Product.aspx?Item=N82E16816111045 and I just add hard rives as needed. The laundry room worked best for me.

Re:My setup

[Electricity Likes Me]

XFS still has some pretty poor default settings I have to say. I've still had some instances where a bunch of files got set to 0 bytes due to a system crash (fortunately just some working files of programs that were open).

I'm finding the tendency of software writers of all stripes to set really unsafe defaults on storage hardware very frustrating.

Re:My setup

[RobbieCrash]

err.. This SATA controller. And woops on the lack of closing anchor on the OI paragraph.

Re:My setup

[inkrypted]

I have come to find that it really depends on the type of data being stored. The file system was really not designed to hold media files but I have found noting better.

No Garage

[WoodburyMan]

I would think about putting the box in the garage. Yes it seems like a great location, it's out of the way and such. However it might not be the cleanest place in the house. I for one know my garage to be one of the dirtiest places. In the winter the car drags in massive amounts of sand from the the winter roads, and leaves in the spring. Spiders and other insects, not to mention baby snakes and rodents, also make their way in from time to time and would just live a nice warm dark place inside the case to live...in city area's it could also attract roaches in from outside. (Despite sonic repellents and traps they still get in). Combined all that with being near moisture (wet car or rainy days). I don't see the case lasting long there. It would need to be cleaned out fairly often to keep fans and heat sinks from gunking up. Of course I understand some people's garages are nice and clean, and not subject to some of these things, but just saying I know for me it would not work out well.

You need a NAS.

[pcx]

Since you're not complaining about processing power or ram, you're in the market for a NAS. There are several good brands. I personally use Synology. It's a bit pricy but you get what you pay for. Personally I'd just add a few external USB drives until the prices fall (they're pretty outrageous now). When prices fall, get a nice 5 bay and stock it with 3TB drives that will give you ~12 TB in raid 5 and ~9 TB in raid 6 (recommended unless you like living on the edge).

You'll probably find the Synology can replace your existing media server (it has a pretty good support community) but since its all linux you can mount the nfs like another hard drive wherever you want the space in your existing drive structure. As your media server grows you can just get another NAS and keep expanding that way as long as you need.

Re:Mac mini or apple Tv

[aaron552]

He wants a media server not a HTPC.

NAS

[hakova]

May I suggest setting up a NAS? Wherever you like (garage, or wherever). You can set it up with RAID, and use it as a backup server too if you would like. This way, you don't need to get rid of your current system but just add up. There are a few of them now with plenty of HDD space and flexible RAID configuration, so that you don't have to get all HDDs with same capacity, specifications, etc. Therefore, you can even upgrade HDDs as you go. I personally have Synology DS1511 in my wishlist, although it is not a cheap toy. I don't feel I know enough to recommend any other file system. ext3,ext4, NTFS are what I use currently and I am naive if I am missing anything. Similarly, I don't know much about the differences between various linux flavors and BSD, therefore I think whichever you are most comfortable with should serve you well. x264/mkv is the format I use too, even for my blueray backups. It serves me well, and I personally don't plan to change it any time soon.

Re:Mac mini or apple Tv

[PNutts]

I was going to suggest Apple as a joke, but rereading it his main issue is storage and he does ask about OS as part of his issue. I don't consider storage and front end coupled so the Mac mini and iTunes is a legitimate interface suggestion as long as he understands the implications to the content.

Re:Mac mini or apple Tv

[pcx]

A mac mini is a perfectly valid (if expensive) media solution. I personally use 2 synology NASs feeding a Mac Mini which is a dedicated iTunes server feeding 4 apple TVs (When I win the lotto I might try out a promise pegasus but until then the synology NASs are growing the library nicely). It's a bit of a pain to remux mkvs to mp4s but it works and it's a really nice solution once you've gone through the headaches of setting it up (yes better than a boxee box -- tried it, now gathering dust, better than every DLNA solution I've tried). The Apple TVs are just really, really nice media end-points and iTunes is a perfectly good management system.

But yea iCloud has no place in any video solution (your purchased shows can be streamed -- movies can not, and this solution will introduce you to your ISP's bandwidth caps very quickly).

I was saying maybe a raid card but non raid card

[Joe_Dragon]

useing software raid is ok most boards have about 6 ports so if you want like 10 then maybe a x4 or better pci-e card may be needed.

Hard drive math and hobbyist heuristics

[Wrath0fb0b]

Ballpark figures, this isn't exact, redo it with your preferred constants, I'm just trying to explain my reasoning against huge enclosures with > 10 drives,

Standard drive idle usage (W) ~ 10W [1]
Low-power (green) drive idle usage (W) ~ 5W [1]
Cost of power ~0.20 $/KWH
Cost of an older drive per year = $17
Cost of a green drive per year = $8.50
Replacing 6x500GB older drives with one 3TB green power savings = $95/yr

So think about that for a sec. At $150[2] for a 3TB drive, you cover the price in power savings in 18 months. That's assuming that there is zero fixed-cost per drive. At the point where you are talking about adding SATA controllers or fancy multi-bay enclosures or, worse, external enclosures with their own PSUs (and fans!), the turnaround-point for older drives is far sooner.

I'm a hobbyist, I understand that it's really cool to make do with older hardware and feel like you aren't letting anything to go waste but sometimes using old hardware instead of buying new is penny-wise and pound-foolish. Spending money on increasing how many hard drives you can accommodate instead of just buying newer high-capacity lower-wattage drives is absolutely batty; especially when you get into the price for anything remotely good in the RAID dept.

My advice, move everything to the largest capacity drives that are reasonably priced (after the flood damage is sorted). Replace the drives when you can do between 4:1 and 6:1 replacement -- should be every 3-4 years. Live happily, quietly and simpler. Save money on power transparently.

[1] http://hothardware.com/Reviews/Western-Digital-2TB-Caviar-Green-Power-Hard-Drive/
[2] I bought some Hitachi 3TBs before the Thailand floods at $130 on Newegg. Of course you would be silly as heck to buy hard drives now for your hobby storage project before they at least fall back to pre-flood level.
[3] http://www.newegg.com/Product/Product.aspx?Item=N82E16817182221
[4] Older drives need not go to waste, they can become offline storage with a simple USB dock[3] -- make a backup, throw it in an anti-static bag, leave it at your relative's house when you visit!
[5] http://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_time.svg

Re:Hard drive math and hobbyist heuristics

[Anonymous Coward]

Yes because nothing says "living simple" like needing every movie ever made on a bunch of hard drives in your living room. The things you freaks obssess over... (rolls eyes)

Re:Hard drive math and hobbyist heuristics

[furbyhater]

He didn't say "live simple" but "live simpler"...learn to read.

my samba server

[Anonymous Coward]

I built a samba server a few months ago. it has 6 sata connections on the motherboard. I'm running it with 6 3 tb hard drives running raid 5 and i'm loving it. it has a lot of space so its very future resistant, and since I chose to go with an amd fusion cpu it is very low power. dual core 1.6 ghz. and the cpu heatsink is fanless, so there's no sound at all. ram is dirt cheap so I threw 8 gigs in there just because it was like 40 dollars.

the amd fusion motherboard cpu combo, 6 3 terabyte drives and 8 gigs of ram cost me like 1 thousand dollars. look around on newegg there's lots of options. hard drives are pretty expensive right now, though, luckily i got mine before they shot up. the one i got also has a pci-express expansion slot so you can easily turn that 6 sata ports into 10 with a sata card. there's no need to buy a server rack.

I'm running ubuntu server just because it was easy, but you could just as easily do it on debian, and if you're a free software/open source advocate debian is probably the way to go since it's running no binary blobs and the open source software is seperate from the core repos. though ubuntu 12.04 is going to be supported for 5 years, so long term that may be the way to go, if you want to wait that long.

I would recommend sticking with h264 though, because even with open source software its very well supported, and it's the best thing out there right now.

at the moment ext4 is the standard for most linux based operating systems, but it's going to be replaced by btrfs in the coming months(years?) once all the bugs are sorted out.

as for media software, I assume you mean on the fly transcoding media streaming software? I would look into ps3 media server or mediatomb. if you don't need to transcode on the fly why not just use a samba server to access your files?

also if you want to shy away from linux take a look at freenas which is based on freebsd, and uses the ZFS filesystem which has built-in raid5-like support, along with about a million other features. just so you know though it very strongly recommends a 64 bit compatible cpu and needs a lot of ram, though an amd fusion cpu with 8 gigs of ram should cover your needs just fine, since you have no use for deduplication.

Get a NAS

[Anonymous Coward]

I went from running my own servers (debian based) to just a simple NAS (ReadyNAS Pioneer is what I currently use). I got so tired of doing maintenance on the server al the time, and the family got tired of it sometimes being in various states of flux as I was upgrading or adding something. The NAS supports the regular streaming and file sharing protocols and updates when you want to apply them are just a button click via a browser interface. When I run out of space I just swap out the old disks one at a time, the server rebuilds the raid on the new disks, and once I have swapped them all out the NAS automatically re-sizes the entire volume. I use XBMC (xbmc.org) for a media client on a variety of endpoints (desktops, laptops, jailbroken ATV2, etc) and it all 'just works'.

I am also going to move mail from my house, probably to a gmail hosted solution because maintaining a mail server and spam filtering etc is also getting old. I still have some systems I can tinker around with, but the rest of the family likes their 'production' systems to just work.

with 8TB data you are looking at 3 3 TB HDD's

[Joe_Dragon]

now just to fit the data on one set is like 3 HDD's and then raid 5 or 6 starts to look like a good idea.

Re:

[Anonymous Coward]

> "I started off building a media library a few years ago with an old PC running Ubuntu. Folders for photos, ogg vorbis music from my CD collection, and x264 encoded mkv movies. I have a high spec machine for encoding, but over the years I've moved the server to a bigger case, with 8 TB of disk capacity, and reverted back to Debian, but still running with the same AMD Sempron processor and 2GB RAM. It's working well,

OK, so it's working well. Why are you fixing what isn't broken? Regarding any CPU choice, I've ran at least a dozen types of CPUs with Linux or BSD on them, and they are all good. I'm fond of the m68040. What I'm saying is that CPU choice doesn't matter. With x86 CPU's your choice is between uber cooling and overclocking versus twitchy overclocking. AMD's tend to be more tolerant of very low temperatures, in case you have some dry ice chillin'. Regarding RAM, it's cheap, buy as much as possible. Fill all the slots, and test the BIOS' limits. RAM has always been my upgrade bottleneck (what I run out of first). I'm mad I didn't do exactly this with my current builds. Currently running AMD because it's cheaper, or so they say.

> it's also the family mail server, and the kids are starting to use it for network storage, and it runs both link and twonkyserver, but my disks are almost full, and there are no more internal slots. The obvious option to me is to add in a couple of SATA PCI cards, to give me 4 more drives, and buy an externally powered enclosure, but that doesn't feel very elegant.

Get a new server! That's what you want to hear. Just find some nice large full tower case (10+ drive bays, easily) and spend as needed on it's innards. Building is the fun part, so try to do something you haven't done. Multiple SATA cards are for sure going to be needed, as well as plenty of drives. Don't forget the spare(s). It has been my experience that running offboard and onboard SATA with it all in the same array tends to suck. I've done it on a few wildly different machines and it's never pretty for some reason. Might just be luck.

> I'm a bit of an amateur, so I'd like some advice.

Hi! You must be new here! Welcome, try the coffee cake!

> Should I start looking at a rack system? Something that can accommodate, say, 10 3.5" drives (I'm thinking long term, and some redundancy)?

Probably not. Racks are good if you can build around them, but for most amateurs it's out of the price range, however with some digging for deals this setup could probably be done for $300 or less, depending on how many servers and how fancy you want to get (12v rails? Solar backup batteries?). Don't expect your existing hardware to work with the cases, unless you don't mind using giant rack cases. Racks are best for when space is a consideration and money isn't. Normal cases are best for everything else. The exception to this is if you have a nice location already handy, like a suitable wired & cooled closet, and you *really* want to use racks.

> Also, what about location — I could run some cat6 to the garage and move it out of the house, in case noise is an issue.

A garage is a bad choice depending on your local climate because heat is your enemy and because of the large ambient temperature swings. You will need to cool this thing, at the least. Noise is never an issue, you're a nerd, loud cooling fans come with this territory. Either get over it, quiet it, or delve in to watercooling.

> Finally, what about file format, file system, and OS/software? I'm currently running with ext3 and Debian Squeeze. Happy with my audio encoding choice, but not sure about x264 and mkv. I'd also consider different media server software, too. Any comments appreciated."

Here at the house I run a Debian shop as well, and also still use EXT3, which is to say that if isn't broken, then don't fix it. I can't comment on x264 vs mkv, but I prefer mkv for no particular reason. I just don't care enough. What do you mean by "media server software"? DLNA is the current hotness,

my setup

[Darkseer]

Price is alway an issue. You are using the space so you should invest in decent storage capacity. I personally spin about 15 SATA drives off of a single raid card.

I recomend the following:
Build a decent NAS head with the following recipe and share it out to all your other devices.

buy a 4U case with ~ 20 drive bays that link to a SATA bus ~$300 (http://www.newegg.com/Product/Product.aspx?Item=N82E16811219041)
                -You may need to mod the case if the port multiplier is SAS instead if SATA
buy a high point RAID controller in the 2000 class or higher (http://www.newegg.com/Product/Product.aspx?Item=N82E16816115049) (32 drives max with 4 port port multipliers and mini sas - 4 sata cable)
Start with 4x 2TB drives (RAID 5, 3 + hot spare)
1 gigabit ethernet hub

then attach it to a cheap cpu with a good amount of ram and gigabit ethernet. Stick the storage kit in the garage and run cat 6. Put the hub in the house and wire it to any machine playing back HD video, for the rest I find you can get away with wireless G.

You are looking at about a 1K investment in kit for storage. For OS I'd use your favorite flavor of linux, gentoo for me but I recomend deb or ubuntu for other people, that supports lvm. This way you can add new drives and expand the storage dynamically. You can expand ext3 or ext4 easilly, you just can't shrink them. In my opinion there no need to go exotic with the FS.

Once you set up the storage share is both via nfs to anything playing high def content and samba for everyone else. Then sit back, relax and enjoy.

External Drives & RAID

[nion]

I'm doing about the same thing, though my overall capacity is much lower due to the fact that I haven't been able to upgrade my 5 750gb drives in a few years. However I went to an external enclosure (Venus T5 from Newegg) and raid5. If the box is in the house, yeah I can see how you might not want an external enclosure next to the tv. However, once you stick it in the garage, that becomes moot. Who cares what it looks like in the garage? Run some cat6 to it, put a gigabit switch on the network (or 2 or 5) and enjoy having a nice, quiet living room. I'm using boxee boxes as the media frontends - one actual dlink box and a pc running ubuntu and boxee.

I personally hope HD prices start to drop sometime soon, I'd REALLY like to upgrade those drives but can't justify the prices right now for the size I'd like (at least 2TB).

Was going to suggest...

[gQuigs]

The ZaReason MediaBox (http://zareason.com/shop/MediaBox-4220.html) but you have clearly outgrown that already... Both ZaReason (http://zareason.com/shop/Servers/) and System76 (http://www.system76.com/servers/) have server models that look like they would meet your needs. I usually prefer to buy from one of them (ZaReason will ship with Debian already). Another thing I would look at is using a video card for encoding. Couldn't find a link for how to do this, if anyone does please chime in. As for filesystems ext4 is the easy choice. If you build it in a year (and have update software including fsck for btrfs) I would go with that. For the software... what are using on the frontend/need to be compatible with?

Re:Was going to suggest...

[gQuigs]

I swear I've changed that setting before...

Dammit.

A better chassis

[GPLHost-Thomas]

If the issue is just plain physical space for putting more HDDs, get one of these Supermicro storage solutions. There's from 15 to 36 HDD chassis, so take your pick. It will cost "only" few hundred bucks. Then yes, you don't want this to sit in your main room, it would be too noisy. If there's no humidity in your garage, and it doesn't get too hot in there in the summer, then using a cat6 to it should be fine. If you don't want to change your motherboard just right away, then just use the old existing one you have currently for the moment, anyway any ATx board will fit in.

Re:Mac mini or apple Tv

[hairyfeet]

If he just wants to go cheap and its gonna be stuck in some garage anyway i'd say white trash it like we did at my last shop. Me and the boss took a couple of the biggest cheapo ATX cases we could get our hands on, took out the motherboard mount and cut the frame on one, and finally a couple of small weld stuck them together. We ended up with something like 16 SCSI drives in that sucker for a total of 2Tb when most folks were still getting 80Gb drives. with those babies loaded with every single driver for just about every piece of hardware up to that point it was quick work to reload a PC.

:As for the board I'd go with something like a cheap AMD board with a nice cheap Phenom low power quad. you can get a Phenom I quad for $55 at Starmicro and add a couple of Gb of DDR 2 for maybe $25, board for around $40 and finally a decent HSF for around $15.

So when you figure in the cases which with no PSU can be had for less than $20 a piece from many places like Geeks you are looking at a final total of around $185 for a quad with 2Gb of RAM with an add on card for adding more SATA slots. Sure it won't be the prettiest thing around but if its just gonna be serving files from a corner somewhere who cares? I'd add a little more and get a full size board with lots of SATA slots and 3 PCI slots for adding more SATA cards and you'd have a thing you could load to the brim and with the low power Phenom quad you'll have plenty of power for controlling the whole deal and maybe even it doing some of the transcoding work via scripts at night when its not serving files.

Serviio for media server

[eshadowrun]

For media server, I recommend Serviio, and not _too_ worry about video encoding that much (on-the-fly transcoding, but a few mkv files do have issues, depending on how they were encoded). And Linux/Windows/Mac version are available for free.

Re:I was saying maybe a raid card but non raid car

[nabsltd]

useing software raid is ok most boards have about 6 ports so if you want like 10 then maybe a x4 or better pci-e card may be needed.

Or, get an actual server board (this is gonna be a server, right?), like this one. That's six SATA ports and 8 SAS ports. If you flash the SAS ROM to the "no-RAID" version, the controller is recognized natively by Linux. In addition, you get lots of PCIe connectivity, a pair of Gigabit Ethernet ports, and IPMI (allowing remote power cycle).

Then, find a full-tower case with lots of 5-1/4" drive bays, and add hot swap bays. There are smaller versions, as well...just budget what you need for drives.

I use the motherboard I referenced along with an add-on 8-port SATA card (anything supported by Linux would be fine) and two of the drive bays for ten 2TB drives in RAID-10. I boot Fedora off a pair of SSDs in RAID-1 and also have four 2-1/2" 750GB drives in RAID-10. The 10TB array serves iSCSI over 10Gbit Ethernet to ESX systems that hold all my VMs, with the 1.5TB array as local and NFS storage. There's still PCIe slots available if you need more controller cards.

With this setup, the VMs are how everything is accessed, so you can pick whatever OS you want to face client machines.

Dedicated unRAID server box

[theyost]

Once your data grows past a certain level redundancy becomes a must. For about 3 years I have been running an unRAID server & I don't know how I survived before it. It allowed me to start with 3 drives (approx 250MB each) to my current configuration of 14 hard drives (mostly 1TB but also a couple smaller drives and three 2TB drives). After you have the system setup you don't need a monitor. I have it in my basement which helps with noise issues that result from having 12 drives spinning 24/7. Because this is a SOFTWARE-based raid-like system I can mix/match different types of drives to get the most storage for the money. I HAVE had a drive fail and the rebuild was easy. Another plus is the data is NOT stripped so if the motherboard ever fails the data can still be accessed (any computer that can access data on a "reiserfs" (I think) filesystem). One recommendation... Wire for Gigabit Ethernet! Once you start transferring HD Video (terabyte-size files) you'll find it helpful. Another recommendation... Find a smaller (100GB-200GB) drive to use as a "cache drive." This will speed up the transfers to your unRaid box :) Good luck! PS1: http://lime-technology.com/ PS2: I have been running version 5.0-beta14 without any issues

Thoughts from my home storage server experience.

[jafo]

I wrote about the latest storage server I built back in 2008, and a lot of my thoughts at the time are written up in http://www.tummy.com/Community/Articles/ultimatestorage2008/

However, to answer a few of your questions...

External disc enclosures? Avoid them like the plague. My initial experience with the 5 bay eSATA enclosures was pretty good -- sometimes it wouldn't pick up the external drives, but usually I could get it to find them after some tweaking, rebooting, etc... I ended up getting 3 of them, the AMS DS-2350S, which at the time were well reviewed, etc... I have since pulled all 3 of them out of active use and have them just sitting around. I don't know exactly the mode of the failures, but eventually after replacing some with others, I finally put them in internal SATA enclosures, which have been very reliable (I used these Supermicro CSE-M35T-1.

Also note that eSATA connectors don't really hold on that well. If anything, they're not as robust as internal SATA connectors, despite being outside the case where they can get banged around.

If I were to do it over again, I'd probably stick with the case I started with, with 5 internal 3.5" bays, and 3 front 5.25" bays, and put the Supermicro in there. I'd also probably go with fewer big drives rather than more smaller drives like I did previously (even though at the time the drives were free, I had them from another project).

As far as running it in the garage, don't even think it, unless your garage is not where you store your cars. I have some computers that I've run in the garage for the last 9 months, and they are filthy, I've had a lot of fan failures, lots of dust, insects, and random other crap. I put mine in our furnace room, which has enough extra space.

As far as using a server case? Hard to see the payback there unless you have a cabinet. Most server cases are HUGE, heavy, and expensive. A 3U case with 12 drive bays likely costs $500, plus you usually have to deal with special form-factor power supplies, expect to spend another $200 on one of those. I wouldn't do it, and I have a 3U 12-bay Chenbro case just sitting at my office that I could re-purpose.

As far as the file-system, I selected ZFS (via zfs-fuse under Linux) and I've been VERY happy with it. The primary benefit is that it checksums *ALL* data and can recover from some types of corruption or at least alert about corruption if it can't correct it. So, if you are storing photos or home videos that you may not be accessing very often, that's good peace of mind to have, I know in 10 years I won't go to look at some photographs I've taken and find they were silently corrupted. Of course, you could get similar benefits by saving off a database of file checksums and checking and alerting if they are bad. Really the only downside of ZFS that I've seen is that if you need to do a RAID rebuild it is a seek-heavy task rather than just streaming. I have a 8x2TB drive array that I'm currently rebuilding (drive failure, at work), and it's 33% done after 31 hours. A normal RAID-5 array would have rebuilt that in what, 10? The system is idle except for the rebuild.

If you care about the data going into it, make sure you checksum and verify the files regularly.

The 8 port PCI SATA card I got is fantastic, it's a Supermicro with the Marvel chipset and is very well supported (even supported by Nexenta).

Finally, all this data is encrypted, so if someone were to burgle us I only have to worry about them getting the hardware, I don't have to worry about them now having scanned bills and other documents and other personal and private data, etc... This is why I'm running ZFS in Linux, it gave me encryption plus ZFS (not available otherwise in 2008), as well as being an OS I'm very familiar with.

As far as OS, I am personally running CentOS on my system because that means I can install and set it up and then forget about it for quite a few years, except for regularly running "yum update". Debian should be fine, but you will get/have to track upstream changes more frequently.

*Not* RAID 1

[Theaetetus]

I went away from RAID 1 for my home server - mirrored drives are great when you want exact copies of data with a backup in case of drive failure... but that's not precisely what I want - I want the right copy of the data. If the main drive is corrupted, then I don't want that corruption to be copied to the mirror drive. Writes can be verified... retained data, less so.

So - and feel free to replace this with your OS of choice - I went with OSX and Time Machine. The main drive stores all of my audio/video/picture files, and I do incremental backups to a second drive. If a file is corrupted, I can roll back a week and restore. If the whole drive dies, I can do a full restore from the backup.

As for other levels of RAID - the biggest bottleneck in your system is probably going to be your network, not your hard drive read speed... Though we'd all love to have RAID 5+1 systems, there's really no need if it's a media server... Realistically, you're only serving one stream of data at a time, maybe two, but that's it.

Re:Mac mini or apple Tv

[sortius_nod]

Just because you don't understand the request doesn't mean it's gobbledegooke, it means your knowledge is limited.

Personally, I don't trust any auto-encoding solutions as they easily go haywire. I'd suggest doing that all by hand.

Ext3 is fine, & a rack mount is a necessity. If you want smooth operation of the system, at least 2 network cards are a must (I run 3, 2 bonded for media/SMB, 1 for management & VPN). I'd suggest having a decent 16 port switch in the house & running the 2 (or more) cables to the box.

For DLNA I just run miniDLNA & for torrents I've just set up uTorrent with a web interface. There's very little my desktop actually does.

you haven't looked at the MKV spec

[Anonymous Coward]

MKV may be "the best" if you're looking at a list of features, but if you had looked at the actual spec for the file format you would know that it is more complex and convoluted than anything imaginable. Seriously. The MKV spec makes a human DNA sequence look like a hello world program.

Re:you haven't looked at the MKV spec

[lister king of smeg]

Knowing a thing or to about genetics not much short of string theory makes Dna look simple

Re:Mac mini or apple Tv

[BLKMGK]

Ummmm, why two nic? The HDD and it's bus are the bottleneck - I cannot max out a gig-E nic now on my server but can play 1080P via x.264 with surround sound DTS encoded on a 100meg connection. No stuttering, no issues, no muss no fuss. It's wired for gig-E but sadly this nic refuses to synch at it.

I WOULD run some sort of redundant storage. However I wouldn't go traditional RAID. Striping data across disks along with parity makes for lots of speed, also means the disks never shut down. No thanks! I happen to use unRAID from Lime Technology. Parity isn't striped, it's held on a single disk. Each disk uses a standard format - ResierFS (ick). But's journaled and standard enough that recovery is easier. On top of that I can pull a disk and get data from it on another machine - one disk removed I still see all my data. Last but not least - if I lose multiple disks at once I only lose the data on THOSE disks and not the entire thing. Losing one disk I lose nothing. In 6+ years or so I've never lost more than one disk. Since parity isn't striped and neither is data one of my servers has ALL drives spun down and quiet, the other has just 2. Between them I have a bit over 22TB worth of disk BTW and not all are 2TB disks but as I fill smaller ones I swap in 2TB disks, the system migrates the data fine. Other programs can be run on this system, it's Linux based, but it's trickier than a full OS install - which has been done by some too.

Honestly to me it sounds like his current setup is working save for disk space. Why not just upgrade to 2TB disks? Surely he has more than 4 ports? If add-on cards are needed they are plentiful. I would also suggest using 4n1 cages for easy swapping. A low speed CPU is fine, underclock it maybe too - HDD bus is the bottleneck!

I do it with my NAS

[houghi]

I bought a QNAP NAS. It runs Linux, has media server stuff pre-installed and runs Linux. You can even install extra software if you like or just trow out all the software and install a "real OS" (Their words) like Debian.

http://www.qnap.com/pro_detail_feature.asp?p_id=134 shows a list. Connecting my Linux machines goes over NFS. If I had a MAC, I could use AFP and for Windows there is Samba.

The standard possibilities are almost endless and as you can install extra software, they really are limited to your imagination and knowledge.

Re:I do it with my NAS

[JayAEU]

Very true indeed. On top of the great set of features already included, it's easy to enhance those things with extra packages. It's just so much less hassle to maintain these things if your main job isn't being a sysadmin. Anybody can basically do it and in a fraction of the time needed for regular server management, too.

Some might argue that these NAS things are too expensive compared to self-built systems. If you consider the time spent on them however, NAS systems beat DIY-systems hands down.

Get a NAS

[JayAEU]

As many others have already stated, a NAS definitely is the way to go here. There are 2 good manufacturers that accomodate any need and have vibrant communities providing excellent support on top of what the manufacturers themselves offer: QNAP and Synology.

Both of them basically use custom Linux builds on their otherwise very PC-like hardware that is open to all sorts of tweaking and readily allows for adding all sorts of extra software.

encoding

[hellop2]

You seemed unsure about encoding. I suggest h.264 video and AAC audio, since it's the only thing supported by Flash, Silverlight, iOS, Android.

Re:encoding

[cynyr]

lets discuss h.264 and Profiles and Levels

One not all devices support all profiles and levels. For example my ps3 only does profile main or high level 4.1 or higher when in mp4 containers. My phone doesn't support high profile, it also doesn't always like my variable bit rate files, as it has a bit rate cap.

I wish manufacturers would better qualify which parts of the h264 spec they support.

Re:Mac mini or apple Tv

[iamhassi]

He wants a media server not a HTPC.

then pretty much any old PC with a bunch of 3.5" drives would work.

I googled "build a media server" and found several guides. Here's one for $300 with six 3.5" bays and here's a 4U rackmount server case with twenty 3.5" drive bays for roughly $1,000

Re:Xbox Media Server

[BLKMGK]

XBMC however on the old hardware it's not going to do high bitrate HD. Better to go ION hardware or an aTV with XBMC, it's been ported to Linux, Windows, and OSX - Plex is based on it.

This!! ^^^^

[BLKMGK]

Except with unRAID, sorry I don't need enterprise RAID at home but the hardware you chose is solid and the 5n1 adapters rock! I can stream 1080P video with DTS on a 100meg NIC, I sure don't need RAID 5 or 6 spinning my drives all day, let them spin down :)

Re:Mac mini or apple Tv

[AmiMoJo]

Atom based boards are pretty good. Make sure you get one of the 5xx series, the dual core hyperthreaded ones. Earlier versions have terrible performance but the 5xx cores are actually pretty good. Won't win any benchmarks for computationally intensive tasks but for responsive file serving and some background downloading they are ideal. I have an Intel mobo with the CPU soldered one and it draws about 11W including the HDD and 2GB of RAM most of the time, just be sure to get a low power DC PSU rather than the usual generic ATX.

Mine runs Windows Server 2008 Free Edition (by extending the trial period repeatedly using scripts you can get about 9 months per install for nothing, and then you only need a quick re-install and some more scripts to start fresh and restore your apps/data).

Re:Endless Storage Expansion

[Keruo]

I've never used Openfiler but I agree with expanding the storage with iSCSI, though iSCSI traffic should not be in his local network.
Each disk node should have dedicated 1Gbit connection without switches between the master server and the disk node.
If direct links are not possible and switches are needed, those should be manageable and configured with iSCSI in its own vlan.
Also important, unbind all other services from the NIC dedicated to iSCSI.
The disk node can be connected to the local network using 10/100mbit card for management via ssh etc.

Re:I was saying maybe a raid card but non raid car

[sirsnork]

You do know, any ROM flash is just software raid right? Even SAS controllers don't have hardware RAID unless you buy a real raid card for $$$. Real raid cards have write back memory and a BBU.

That being the case you're better off using something like linux software RAID so if your SAS controller happens to die, you can still recover your array by simply plugging it into another machine

Define the problem a little bit better

[drolli]

a) i am running out of hd space

b) i feel (=movies dont play without interrupting) that the processor is a little slow
c) i am bored over the christmas holidays

d) i am worried the thing explodes or falls apart

if only a): attach a NAS and make an archiving system

if a) and b) or d): there are enough shopping guides for off-the shelf pc's out there. You priority should be energy consumption, reliablility and space for more hds. Condiser external sata boxes

c) play around with something differnt.

My tips

[DeBaas]

My tips:
Hardware
- Get a tower with a lot of 5,25 inch slots (there are midi towers with 9 slots available)
- Get passive harddisk coolers that you screw on the drive and then need a 5,25 inch slot such as: http://www.akasa.com.tw/update.php?tpl=product/product.detail.tpl&no=181&type=Cooling%20solutions&type_sub=HDD%20Cooler&model=AK-HD-03BK
- Get a highly efficient PSU (I got good results with Enermax 87+ series)
- Get 1 or 2 silent and high quality 12cm fans
- Get a CPU cooler that is mostly cooling block with on top a 12cm FAN

I did this and have a system so silent I need the leds to see that is on. Also allows for massive amounts of storage.

For software, you may want to take a look at http://www.nexentastor.org/projects/site/wiki/CommunityEdition
I don't yet have personal experience with this, but it is often used now in professional hosting.

And then put it in the basement and use something like a popcorn hour to connect to it over NFS to use the content :-)

Re:My tips

[cynyr]

I would recommend the fanless (or the fanned versions) Seasonic PSUs. Totally fanless, 92% efficient, low ripple output.

http://www.silentpcreview.com/Seasonic_X-400_Fanless_PSU

There are some fanless ATOM and Fusion based mini-itx broads around with PCI-E slots around as well. SPCR has some good tips for making things quiet as well.

You don't need a rack system for 10 disks

[Lazy Jones]

There are plenty of good, quiet PC tower cases with 10 disk slots nowdays, for example the Fractal Design Define XL. Noise should not be a problem this way, if it isn't with your current solution. Migrating your old board into such a spacious case should not be an issue.

Personally, I would try to avoid more than 10 disks for home use because it'll become a hassle replacing the defective ones at some point. With 3TB SATA disks available at the moment, you can get around 24TB with enough redundancy (RAID-5 with 1 spare or RAID-6; "enough" meaning that you don't have to switch it off when 1 disk dies until you can replace it). No redundancy i.e. RAID-0 should be out of the question if you value your data.

To connect and handle the disks, you can look for a SATA RAID controller like the 3ware 9650SE-12ML. It's not cheap - noticeably more expensive than 2-3 additional plain SATA controllers, but easier to handle than software-RAID and bootable (Linux still needs an ugly workaround to boot from software RAID-5). On the other hand, I found it easier to migrate to bigger disks one-by-one using software-RAID than with various (expensive) RAID controllers, but YMMV.

i used esata

[peawormsworth]

Perhaps its not the greatest solution... but I suggest that adding esata drives is easy. That is what I did when I ran out of drive space on my server. Install an esata card and buy one esata drive at a time. That way u get maximum space at minimum price as you buy new drives as required. It does not require any changes to ur existing setup. And esata is just as fast as an internal hard drive. This saved me a lot of time, because the software is setup like a Tivo to auto archive all my favourite shows... and some I dont watch, but keep for others. This means reconfiguring the system is much more time consuming then just reinstalling Linux on a new box. So I just renamed the existing archive directory name, and symlinked that name to point to a directory on the new drive, copied all the subfolders where all the shows reside add configured the tv software to access the old shows through their new directory name. This was fast, painless and not too expensive. Aside: To the person who suggests u run compression manual on each show... well they are not archiving 30 shows per day and have no concept of what a computer is meant to do... ie: automate beyond anything u could possibly do by hand.

Re:Mac mini or apple Tv

[wcb4]

i disagree that itunes is valid for managing media. I have triedon numerous occassions to let it catalog my media library and it just chokes horribly. admittedly, not everyone has 65,000+ music files, but iTunes can't handle very large collections without barfing horribly.

Google music ...

[giorgist]

In the context of Music, I love google music.
I upload my songs, I clean up the metadata on line, I upload the album art work and it just works.

My android phone instantly gets the updates and correction. If I loose the phone, my next one has the whole library before I leave the store.
I guess Google TV has the same effect, for the desktop. I hope a desktop client comes out.

It is not quite what you are asking, but it may be part of what you are asking.

Format

[Le Fol]

For the FS format, definitively go for ext4. It shall save you a huge amount of time when fsck, it's easier to undelete from. For a media server usage you don't even need to understand the old delayed sync controversy, it's not relevant. For films, you're certainly right with Matroska. For the rest I wont comment (I use x264 myself).

Re:Mac mini or apple Tv

[TheGratefulNet]

unraid sounds just like what I was *designing* for myself (doh!).

the idea of lots of disks always spinning is just moronic for home-users! we are NOT a data center! the game is different, here; and it has taken the industry a while to learn this.

I had a gazillion disks spinning in big-time home raid. for years. blech! lots of noise, heat, failure modes and like you said, you lose too much, you lose it all! the idea of being able to take a single spinner out, mount it on a 'dumb' system and read it, that's super powerful! don't discount that, folks! its worth its weight in gold.

my approach is to just be an idiot human (I do that part well) and save lots of copies of the files to lots of places. I do that anyway. then run a database job that will traverse your filesystem, get every file's size, date, md5 hash and any other tag info you want to peek inside, for. and run a smart differ on it. keep 'n' copies; report partial copies; trim the ones that are beyond what you need (if you want 3 copies, kill any 4,5,etc version you see). use ANY format you want for the disks, ntfs, ext3, jfs, whatever. each disk has a disk-id ('blkid' on linux shows this easily) and so each file has a disk-id to show where its located.

the final part of this is to keep all disks spun down and have the database know which disk the file is on (the one you want to watch/play) and have it do power-mgmt (as I call it) and spin the drive UP. either have it auto-spin down or spin down via mgmt when your 'session' is over.

spinning the disks is easy if they are all external esata/usb/fw. even simple x10 powerline relays can do this (my first proto will use this style of distrib tech). for internal drives, you power their molex's on/off via relays, via software control.

noise is stupid! power/heat wastage is stupid! raid is stupid for most of us. lets get beyond the gazillion spinners and be smarter about our large disk collections.

(oblig: I'm using arduinos and linux as the controller on all this)

at least a 2-box solution

[TheGratefulNet]

my front-end is a fanless asus atom (they only make 1 or 2 of those; get one with an ION video system on it). this works VERY well with (sigh) win7 and win media player. its glitch free. it replaced my popcorn hour since it bettered it (I really hate saying that but its true). also, the linux driver is about 80% as good as the win version but its sufficient for myth-frontend use! I dualboot this asus ion fanless system and go into linux when I want to watch mythtv (works well for live and recorded). it auto powers itself off and even an rf wireless dongle keyboard can wake-up this sleeping system (both lin and win7). its a winner and fully fanless.

the back end can be an i3 (my choice) or even another fanless box (just bought the asus fanless AMD e350 mobo. very nice and good price!) i3 is a good performer and low heat. the amd cheapie cpu is fanless but nowhere near as strong and its borderline for a server. its current linux driver is not impressive for video use so this is still mostly a business-level workstation, not a front-end. but ok for music (slimserver) stuff.

a back-end can be non-silent, so the i3 is my choice. I have a mini-itx i3 system that is tiny and fast enough for most things. and being an i3, its no POS and can do rebuilds very quickly. I like that; and times when I underspec a server, I hate it when I have to do source level builds or updates on it!

Re:Xbox Media Server

[Killjoy_NL]

This is what I did, I built a tiny HTPC house in an old NES case, the only moving parts are in the slot-in dvd burner.
It's an Asus ION3 Deluxe motherboard with 4gb ram. I added a 60GB SSD and installed Ubuntu/XBMC on it, it works perfectly in full HD.
There is no active cooling, but the temperature doesn't get high at all. The deluxe version of the motherboard had an USB IR receiver and remote control which work just fine in XBMC :)
I keep my media on a nice NAS with 4x2TB Drives in it, if that is full, I will add another NAS.

I posted my project for a bit here: http://forum.fok.nl/topic/1470800
The pic with me and the glasses was after I got a molten piece of plastic in my eye, the site is in dutch.
Enjoy :)

RAID is a tool, not a solution

[Anonymous Coward]

Just flipped thorugh the comments. *NO ONE* mentioned backup or failover. Any RAID set you build is vulnerable to drive failures, and vulnerable to catastrophic failures of the RAID controller toolkit or simple "rm -rf /" deleting vital material. And this happens too often ins "built-it-myself!" environments, where the factors that the "high priests of IT" worry about tend to occur in their worst combinations. From people who like to do everything as root "because they know what they're doing", to people who slap dumpster found components together "because it's free!" to people who blithely ignore basic system and security updates "because they trust the people they're working with" or "they have a firewall!", I've seen about every combination of hardware blown, disk drive scrambled, irreplaceable software screwed up by badly written rootkit kind of failure.

If you don't have backups or a thought out failover plan, expect Murphy to camp out and plant a little flag on your virginal backside saying "for sale, no money down on labor day weekend" for every problem that wants to take a ride.

Re:RAID is a tool, not a solution

[MachineShedFred]

Agreed. I have a FreeBSD setup using ZFS. I have it configured for raidz2 (two parity drives) and use zfs snapshotting to do incremental backups to a second raidz1 pool with cheaper drives configured to spin down.

Yes, if the house burns down I'll still lose everything, but this isn't enterprise disaster recovery I'm going for.

Re:Mac mini or apple Tv

[OneMadMuppet]

PCI SATA cards are a really, REALLY bad idea if you have any care about IO performance, and especially if those drives are part of a RAID. Use PCIe SATA cards.

Re:Mac mini or apple Tv

[Whatchamacallit]

http://plexapp.com/ - Plex can run as a media server on your Mac Mini, can serve other Mac's with media. Can also serve to iPads and you can serve to an iPhone or use it as a remote control. It handles MKV and just about all other formats. It's like Boxee in some ways but much much better. It also works with AppleTV's although you might need to jailbreak them if you won't want to use AirPlay. But since you are using AirPlay now, it's no different with Plex. Once Jailbroken, the AppleTV2's have a Plex menu added to browse the media server library and make choices. There's a lot more to it. Plex is free except for the iOS / Android apps.

Re:Mac mini or apple Tv

[Whatchamacallit]

iTunes for Windows? Because iTunes for Mac is quite a bit different. i.e. it works much better. The Windows port is a bit of a hack in my opinion and doesn't work very well.

Re:Mac mini or apple Tv

[Cuisinart]

Curious, just how do you use NAS to feed an iTunes library? I haven't figured out how to elegantly and reliably do this. I've considered using an AFP share and a symbolic link from the local drive to point to the AFP share, but that doesn't seem totally reliable across reboots. I've also considered setting up an iSCSI initiator on the Mini and then mount iSCSI volumes from a NAS server but that also doesn't seem terribly reliable. This has to be reliable enough that when I'm traveling my wife has no problem getting to videos for the 3 year old :-). Please share your setup. Thanks!

Re:Mac mini or apple Tv

[garaged]

In my best times I had >80,000 mp3 on my library, that worked perfectly on amarok, some 7 years ago

Re:Mac mini or apple Tv

[GNious]

I bought an Atom board from Asus - the On-board fan is driving me mad, and apparently every harddrive I connect to it reports SATA Link Error (or: Linux* reports it for every harddrive).

At this junction, I'm contacting the vendor to see what alternatives they have, but Atom is pretty far down my list. Wouldn't mind something based on ARM or similar, since it really is just a file-server with a mirror.

*: Ubuntu Server LTS 10.4 incl backport kernel

Go Low Power / External

[neorush]

I'm not so sure I agree with the summary that "an externally powered enclosure, but that doesn't feel very elegant." There are lots of fantastic external enclosure systems now. A few posts here talk about how external enclosures and RAID never spin down. This is wrong. I have an 8TB (5x2TB) Raid 5 tower (SansDigital) that spins down after 60 mins of inactivity, and takes about 5 seconds to spin back up when someone wants a file. I use an intel dual core atom box (nvidia ion gpu, plays HD just great!) that is smaller than my cable modem with Ubuntu and XBMC. It sits on top of the RAID tower and you barely notice it. With the speed at which my family gathers data I will need to add another enclosure in about 14 months. The next one will be 10x3TB drives (assuming another better box doesn't hit the market) and should last a few years. When I get this one I plan on purchasing another Atom based box and moving it and the enclosures to the closet / leave one atom box hooked to the TV. Ubuntu / XBMC will mount the FS across the network. From my experience most of my family is using wireless (me being the one exception) and the RAID array can read at ~100mb/s and neither the network nor the RAID is ever maxed out even with 3-4 users streaming HD. I am the only one that ever comes close to saturating the network, but my desktop HD's are slower than the gigabyte connection (~60mb/s). I could spend more money on faster hard drives but when I can copy a couple gig file in a minute or two its not worth the cost.

Re:Energy

[jones_supa]

Another option could be to use 2.5" disks as they suck many times less power. Before the floods the Samsung 1TB one was on sale for 90€. I recall there being mounts which allow you to put two of them into a 3.5" slot, too.

Re:Mac mini or apple Tv

[macs4all]

A mac mini is a perfectly valid (if expensive) media solution. I personally use 2 synology NASs feeding a Mac Mini which is a dedicated iTunes server feeding 4 apple TVs (When I win the lotto I might try out a promise pegasus but until then the synology NASs are growing the library nicely). It's a bit of a pain to remux mkvs to mp4s but it works and it's a really nice solution once you've gone through the headaches of setting it up (yes better than a boxee box -- tried it, now gathering dust, better than every DLNA solution I've tried). The Apple TVs are just really, really nice media end-points and iTunes is a perfectly good management system.

But yea iCloud has no place in any video solution (your purchased shows can be streamed -- movies can not, and this solution will introduce you to your ISP's bandwidth caps very quickly).

I will agree that a Mac mini makes a wonderfully stable, no-fuss media server. I set up a combo media-server/htpc system for a friend, based on a HDMI-capable Mac mini, Boxee (went through SEVERAL media servers, first on Windows and then OS X, before deciding on Boxee and the Mac mini), with an IRTrans IR Transceiver and the most-excellent iRed "virtual remote" software. Oh, and all of this is supported by a 6TB Buffalo NAS (already purchased. I would have spec'ed something different).

With this setup, I haven't had to transcode ANYTHING, and this guy had a mixture of every audio and video format the bittorrent world has to offer; so, not having to transcode 50% of his collection just to use iTunes (which I would have preferred to use) was a definite "plus". He also had a requirement that he be able to browse his media by his already-established directory structure (rather than by genre, artist, etc.), which of course is not really an option with iTunes (and getting harder to find in most media servers in general), so Boxee was the best (and most stable!) bet.

However, this system is able to stream audio/video to anywhere in his house, as well as control his legacy A/V gear (which (audio) is already distributed to several places inside and outside) through the IR scripts I have created. iRed has a wonderful built-in webserver that can publish any "remote" you create as a web page; so "anywhere" control becomes as simple as any device that can pull up a web page.

This system has been up and running in one form or another for two years with no major issues, and in its current form for nearly a year. The only remaining issue seems to be finding a good web front-end for Boxee. All the ones I have found have had one fatal flaw or another; but I haven't seriously looked for nearly a year. Currently, there is not really a need for supporting Apple TVs; but if that becomes interesting, we may have to switch to an iTunes-based system, or hope that someone figures out how to install a Boxee-compatible (DLNA?) client on the ATV2.

Since this guy travels a bit, the Mac mini also provides a very secure and robust "gateway" for his to access his files (and media) while on-the-go using SSH/SFTP. He also runs his mail through the mini, and I set up an experimental WebDAV server on the machine as well. Administration of the 'mini is done through Apple Remote Desktop, or through SSH, or, if you're "local", using his TV and his MacBook Pro or a wireless keyboard/trackpad combo.

Re:Mac mini or apple Tv

[macs4all]

Curious, just how do you use NAS to feed an iTunes library? I haven't figured out how to elegantly and reliably do this. I've considered using an AFP share and a symbolic link from the local drive to point to the AFP share, but that doesn't seem totally reliable across reboots. I've also considered setting up an iSCSI initiator on the Mini and then mount iSCSI volumes from a NAS server but that also doesn't seem terribly reliable. This has to be reliable enough that when I'm traveling my wife has no problem getting to videos for the 3 year old :-). Please share your setup. Thanks!

Using a NAS with a AFP share that is defined in the account's Login Items seems to be quite stable across reboots, at least with Snow Leopard. You can add a mounted volume to Login Items, even though the dialog text makes it seem like you can only add "Applications".

Re:Mac mini or apple Tv

[fingon]

Heh, sounds like lot of work. Working as a reasonably well paid consultant, I don't really want to do much hacking on my free time, but I wanted a quiet setup as well.

My approach: Mac Mini + SSD main machine.
Secondary storage: FreeBSD ZFS + RAIDZ2 (SSD cache drive + couple of spinning disks) NAS for video + backups of Mini.

As I don't deal with video much, NAS is mostly turned off, and all I need is one WOL packet to fire it up and mount on the Mini as needed. Win-win.

(Ok, other hardware in the house like gaming PC, laptop, etc also can access NAS as needed ..)

Re:Mac mini or apple Tv

[Sponge Bath]

You won't regret it. Mine works great as a media server and HTPC using Plex. It's attractive, small, silent and easy to setup and maintain. I added a 1TB 2.5" USB drive for media storage. I ended up getting a bluetooth keyboard and magic pad for the coffee table, though between the iPad remote app and a harmony universal remote that isn't really necessary. As an added bonus, I purchased the G-Force and Aeon visualizers for iTunes which look great on the TV while playing music. Everything is backed up automatically via Time Machine to a remote 3TB drive.

I am totally satisfied and am still amazed at how far we've come from the old days of manually perusing through a collection of optical media to select something to listen to or watch.

Re:Mac mini or apple Tv

[Cuisinart]

Yes, I do that today. Though for iTunes use it's a good idea to put a delay in the auto start of iTunes to make sure the share is completely ready. What I meant about reliable, and didn't explain very well, is that iTunes over NAS seems to be slow and klunky according to what I've read on the net. Ripping and importing a CD, for example, can take FOREVER. And syncing content to say an iPhone from iTunes running off a NAS can be verrrrrry slow too. So I was curious how the OP got around this or if he even saw any of it.

Thanks!

Re:Mac mini or apple Tv

[jedidiah]

> I don't really want to do much hacking

The frontend decoder box is not the "hard" part.

Troll harder next time.

Once you get beyond what you can store on a couple of disks, things become a little more interesting. Your suggestion does NOTHING to address that.

And here I was foolishly expecting to be presented some useful alternatives in terms of larger RAID and NAS devices. Silly me.

something similar

[ixidor]

I am currently doing something similar. I decided to go with a sas controller. http://www.servethehome.com/intel-sasuc8i-lsi-lsi-sas3081er-lsi-1068e-based-raid-controller-review/ this will get me 8 drives now, plus ability to add in the sas expander and easily go to 24 drives and more. Im in a case that can do 8 internal drives (plus a 4 in 3 if i choose) for now, http://www.newegg.com/Product/Product.aspx?Item=N82E16811146075 and will probably move to a norco case when i need to expand. something like http://www.newegg.com/Product/Product.aspx?Item=N82E16811219038 as far as fileformats, i say go with what works for you. handbrake and mkv do well for me. server software, i am currently trying to decide between ubuntu/centos or freenas. freenas will let me run from a flashdrive, the others will be easy to do bittorrent, LAMP stack, and run GUI and use file server to actually play movies to the tv. went with a gigabyte mobo and an i3, so it can decode even a 10gb movie and keep up on the bitrate just fine. it mostly just comes down to personal choice at this point, both would work well. just please consider staying away from the windows home server stuff, to many quirks for me. mobo/cpu ram. is a matter of choice. a 6 core phenom in a server chassis in the garage with proper ventilation should do just fine. or an i3/i5. although a friend of mine has windows 7 on computer near his tv, and he says handbrake refuses to work with the i3. but it worked fine when i had it with ubuntu ymmv. at that point it is just a matter how much do you want to spend, how long to wait for the recodes. unles you wanted to spend with abandon, then maybe a supermicro mobo with ipmi or something like a dell and a DRAC card "might" be worth it, since it might live in a closet or garage.

Re:Mac mini or apple Tv

[jedidiah]

Apple TVs are weak and limited. Chances are that if you already have 8TB of stuff then you have lots of stuff that's not playable on the Apple TVs. You don't need to go with an Apple product to enforce the same interface. You can do that with anything.

That 8TB of legacy data also makes any "cloud" solution problematic.

Probably the wrong approach for this kind of user. Might work for someone starting completely from scratch though.

Re:Mac mini or apple Tv

[Patch86]

This is probably a lot to ask of Slashdot, but maybe you should only reply to "Ask Slashdot" questions if you actually understand the question?

iSCSI

[Vrtigo1]

Why limit yourself to a case that can physically accomodate all the drives you will ever need from now until the end of time? There is an easier way. Buy yourself a small form factor PC such as a Mac Mini or roll your own miniITX system. Boot from flash and access storage via iSCSI. In an enterprise setting you may say it's not fast enough, but for home usage it's more then enough. There are lots of manufacturers building iSCSI connectivity in these days, so you could get somethink like a Drobo, etc, or you could roll your own.

Re:Mac mini or apple Tv

[cynyr]

When I was using iTunes, it as only for media player things. Ripping was done from linux. I didn't own am iPhone/iPod/etc then.

Anyways, drop the apple gear and it will work.

Amahi?

[keith_nt4]

I've been mostly skimming this thread but I don't think anybody has pointed to Amahi: seems like at least software-wise it would cover all the requirements for media storage and add/removal of HDDs. I should mention my favorite forum merely recommends Amahi as a FOSS alternate to WHS but that I have not actually used Amahi.

Also, I thought XFS was supposed to be a good/possibly best file system for large files?

raspberry pi

[FlashBuster3000]

Waiting for some decent raspberry pi box.. actually, just someone packaging that nice little board into a nice case.. after that it's all linux+xbmc (again) :)

http://www.raspberrypi.org/

Re:Mac mini or apple Tv

[Lisandro]

Ditto. I have a dual core Atom setup on my mediacenter ( http://www.newegg.com/Product/Product.aspx?Item=N82E16813500036 ), which also doubles as a server for files, backups, HTTP, FTP and Deluge. I can run XBMC on top of that and never had any performance issue whatsoever.

how I'd go about it

[CAIMLAS]

Since I'm constantly looking for something in this department of late, let me offer some input.

* I'd go with an i3 system with a small amount of ram (4GB would do the trick). Make sure the board has at least two PCIe slots which will be suitable
* Intel Ethernet is crucial for network performance, though there are some cards which are similar to the desktop stuff. If network perf isn't an issue for you now I'd not worry too much about it.
* Rack systems are nice, but unless you've got a rack and other rack equipment, in an area you can close off, they're a pain. The fans are often loud due to being small and high RPM, and if you need to power things off and move them around, it means taking whatever is on top (monitor, books, other systems) off (whereas taking the side panel off a tower is fairly straightforward).
* See if you can find a tower with many external 5.25 bays. You can then use something like this to adapt it as hotswap, if you want: link to product. The alternative
* You'll need SATA expansion above and beyond what the board has if you're going to fill the chassis. (The next-generation Sandybridge-E boards may offer this soon, reportedly having up to 10 SATA ports - but that doesn't exist now, and it's sure to demand a premium.) Don't use the Supermicro AOC cards, they've got shit Marvell controllers. Look at picking up some older LSI-based Intel or IBM SAS cards with SFF8087 breakouts for about the same price ($100-125/ea) supporting up to 8 disks/ea.
* Filesystems: I like to have an independent root and keep my storage on something else. Eg. two disks in a mirror and the others in arrays.
* I am a big fan of mdraid on Linux, and hate most hardware RAID implementations. MDRAID works well and is easily managed, enabling you to reassemble the arrays on any other hardware you've got available. ext4 still has its problems, as I understand it, but that's why we have backups. I've had good success with xfs and have been personally using it since around 2000 with good success. Some people like lvm, but I'm personally not a fan due to performance and management headache ( as well as seeing it break entirely too many times and how it encourages poor planning).
* H.264 with MP3 audio sounds like a good bet to me. Alternative to the MKV container is the MP4 container, which has lower CPU overhead for playback and is considered more compatible (eg. you can use it on Apple crap), but there's generally not a necessary reason for MP4. MKV can be repackaged into an MP4 fairly quickly (a minute or two for a full movie), though the reverse is not true (MKV has more b-frame data, amongst other things).

Re:Thoughts from my home storage server experience

[CAIMLAS]

If I may ask, how did you get the Marvell-based Supermicro SAS cards to work well? I couldn't get them from crashing the system after very short use. This was also with fused ZFS on Debian stable.

Re:Mac mini or apple Tv

[Marillion]

Both QNAP and Synology make good NAS units with multimedia capabilities. That said, if you need transcoding for video, don't count on either one having the horsepower to transcode video. They have enough power to saturate GigE for simple file sharing, but you'll need another device with more power to transcode video: (ie: convert a 720p mkv to an iPad friendly format) Out of the box, they're stripped down Linux boxes with Busybox based userland tools. But they have good community based support for extensions from Optware - you can add gcc compilers, perl, ruby and all kinds of goodies to hack your NAS into whatever frankenstein setup you like. The best part ... they're low power consumption.

The tower of storage

[Anaerin]

As the "Anonymous Coward" above has suggested, you can:

Get a full tower case, something like a Lian-Li PC 80 with 12 5.25" bays

Then get 4 5-in-3 hot-swap bays

And as many SATA cards as you need to populate the bays (With 6 SATA connectors on your motherboard, you'd probably need another 2 4-port SATA cards to fill the box)

Top the lot off with a 750W+ power supply, and you're all set. Throw it in the basement, or out in the garage if it's temperature controlled (Would have problems with that up here in Canada; -40 makes for very unhappy PCs without a lot of sealing against condensation).

That'll get you (using 3TB hard-drive) 60TB of storage in a single case. You'll also have 2 free SATA ports, so you could have a separate SSD as your boot drive.

If you're going to be upgrading, you might want to switch to Serviio for your uPNP hosting, as it also handles on-the-fly transcoding into supported formats for uPNP/DLNA hosts using FFMpeg, and upgrading the CPU to enable real-time HD transcoding. Then you can store your media in the best format you can (Native from source, or x264/DTS if the source is raw), and not have to worry about having multiple copies around to deal with client compatibility.

This can all be done incrementally, too, especially if you don't have the funds for it on-hand at the moment (who does?).

Re:Mac mini or apple Tv

[Lisandro]

I have a Zotac Atom/nVidia mobo which works just fine with passive cooling. In fact, that's exactly why i chose it: ttp://www.newegg.com/Product/Product.aspx?Item=N82E16813500036 . Highly recommended.

Only caveat: Use RAID6 not RAID5

[Fencepost]

The argument for using RAID6 (2 parity drives instead of 1) is that with the larger drives (1TB and up) if you have a hardware failure in one of your large drives (probably all from the same lot, possibly with sequential serial numbers and all with an effectively identical environmental history), your odds of having a second failure during a very long rebuild are not negligible. Your example of effectively 1% per hour on the rebuild should scare the crap out of many people - particularly if they've also been sloppy about backing up that 12+TB of data which is pretty likely.

Basically, if you have a pretty full RAID5, replace it with a RAID6 of significantly larger drives - get the extra redundancy, get some growth space, and take that RAID5 somewhere else - preferably preserved as a snapshot or used as a seed for offsite backup if you don't already have a good backup routine.

Re:Only caveat: Use RAID6 not RAID5

[jafo]

I almost mentioned this in my previous post, but didn't. As you suggest, I have great backups, so if the RAID-Z/RAID-5 fails during the rebuild, it's not a huge issue, I just need to drive an hour away to pull down all the data.

In the case of the 1% per hour rebuild, that is actually a work machine with RAID-Z2 (ZFS equivalent of RAID-6). Having that extra safety net of still having the ability for another drive to fail was very nice when I was mucking about with the array.

For home, would I use RAID-6/RAID-Z2? Probably not. But that's because I really, truly have great backups. The other poster talking about using unRAID and how if a drive dies then he only loses the data on that drive, I was thinking "where are your backups"? If you're just storing rips of your CDs or movies, and you're ok with spending the time to re-rip, I guess that's ok. However, I'm storing original content there, I don't want that to go away.

Re:Mac mini or apple Tv

[hairyfeet]

We are talking bulk storage, the guy never said anything about RAID. If it were a RAID sitch i'd agree 100% but I've used PCI SATA for bulk storage and it'll stream movie speed just fine. But considering there are Asrock boards with 3 PCIe slots for like $50 it isn't like you can't go AMD quad and have lots of PCIe if you want, and those include 6 SATA native. So add that in with the $55 quad i linked to you are still right at or under $200 depending on where you get the rest of the parts, for a quad server that's really damned cheap.

Re:Mac mini or apple Tv

[dargaud]

the idea of lots of disks always spinning is just moronic for home-users! we are NOT a data center!

I agree, and I used to have my media server under XP and the disks would sleep unless accessed. But for the last few years I'm Linux all the way (work/home/laptop/media server). And no matter what I try, the disks won't ever spin down. If I send a low-level SLEEP command with smartctl, it spins down and then right back up within 10 seconds. I've researched this problem without solution. Maybe someone here is brighter than me.

Re:Mac mini or apple Tv

[BLKMGK]

Honestly what you're doing sounds like a ton of redundancy and work. I'm also not sure how well HDD will respond to having their power snapped on and off via relays or what that might do to a PC power supply having that sort of loads removed and added. I'd at least look to do something solid state for the switching or better yet use a pure software solution which unRAID is. Having multiple copies of files to preserve them is just as bad as pure RAID or worse from an efficiency prospective with the exception of the fact that you won't lose it as easily.

unRAID isn't perfect but mine have been running almost like an appliance for YEARS with the only changes being software updates, hardware updates, and an occasional lengthy power failure that's drained my UPS. I just checked, last power cycle on one of them was 135 days ago. Sadly I'm also noticing that it's time to upgrade a disk, darn thing is nearly full! :-(

Re:Xbox Media Server

[BLKMGK]

Very nicely done! Chrome had no issue translating :-) That's a pretty good idea and if it doesn't overheat that's even better! I like the little USB controllers, those would have been perfect to integrate. I wish that emulators were a little more easily integrated into XBMC, then you'd be able to play real Nintendo games on the box too!

Very nice project, thanks for sharing it :-)

Umm...

[deesine]

the several months old Sony Blu-Ray player I have plays them just fine. Sorry, don't feel like pulling it out to check the model.

Re:Mac mini or apple Tv

[vlueboy]

Honestly to me it sounds like his current setup is working save for disk space. Why not just upgrade to 2TB disks?

Perhaps you missed the stories this past week about HD warranties scaling back 80% (down to a single measly year), prices being suboptimal until this week, and worldwide supplies all coming from the same compromised source. The heightened risk of expected dampness, environmental dust, and buggy/salvaged/new-but-untested factory equipment at the reconditioned factory makes it better to wait until the re-started post-flood production works out the bugs with some guinea pig other than US geeks.

What is 2, 4, or 1000000TB worth when the disk completely fails? It's more reliable to endure risk of his own equipment's current failure rates for a few more months than to switch to disks of unknown trustworthiness tomorrow. Especially if you're going to christen it with a huge stress on the first night in by transferring the full 2TB collection in one go.

Re:Mac mini or apple Tv

[vlueboy]

Sorry, /. posted under you instead of the GP.
Ugh.

Re:Thoughts from my home storage server experience

[jafo]

The cards I had just worked. Maybe you're talking different cards, I'm pretty sure mine are SATA not SAS (check the part number for more details). I got them originally because they were one of the few chipsets supported by OpenSolaris, and they were apparently the chipsets used in the Sun "Thumper" boxes, but I moved away from the Solaris kernel and found they worked great under Linux as well. Perhaps you have a defective card? I've got maybe 6 of these cards and have been really, really happy with them.

I also played with the Silicon Image cards, and they worked great as well. They're a lot less expensive, but only supported PCI, not PCI-X, and I wanted the extra bandwidth. During RAID rebuilds or verifies, I'm getting 250MB/sec or so, so the extra bandwidth is nice.

Sorry I couldn't offer more help.

Re:Mac mini or apple Tv

[macs4all]

Yes, I do that today. Though for iTunes use it's a good idea to put a delay in the auto start of iTunes to make sure the share is completely ready. What I meant about reliable, and didn't explain very well, is that iTunes over NAS seems to be slow and klunky according to what I've read on the net. Ripping and importing a CD, for example, can take FOREVER. And syncing content to say an iPhone from iTunes running off a NAS can be verrrrrry slow too. So I was curious how the OP got around this or if he even saw any of it.

Thanks!

Good idea on the startup delay for iTunes, especially if iTunes is also a Login Item. Although most of this guy's stuff comes "pre-ripped" (he is a bittorrent addict), I have had the occasion to Import some stuff into his library, and I didn't notice any discernible "delay" or "slowness", in importing up to the NAS. Perhaps the people who have been having problems are running NASs (NASes? NASties?) with a "iTunes Server" built in (many do!), and have the "reindexing" interval set so low that reindexing operations are happening during the import/rip process. Just a guess, though...

Re:Mac mini or apple Tv

[macs4all]

When I was using iTunes, it as only for media player things. Ripping was done from linux. I didn't own am iPhone/iPod/etc then.

Anyways, drop the apple gear and it will work.

I guess you didn't notice my statement that the Mac mini-based system I put together has been working in some fashion for over two years, and with all the current software (hardware hasn't changed) for over a year and a half.

So, you actually got it backwards. Use iTunes to rip to ALAC or 256k AAC, but use either Plex (buggy and unstable) or Boxee (less buggy and much more stable) as your Media Center/Server.

Add boxes

[smash]

Openfiler + zfs + more low end boxes + iscsi target + Nfs = poor mans San. Each box roughly comparable to a shelf of disk. Trying to fit too much in 1 box is going to hurt resiliency and add noise to the single machine to extract all the heat.

Re:Add boxes

[smash]

And by Openfiler I meant Freenas.

Bad Time to buy storage

[maxbash]

I would make best with what you got and wait to buy hard drives. Drives will probably be half the current price next summer. The upside is now is a good time to off load drives. I Ebay-ed a bunch of 4 year old 500 GB drives for about 2 1/2 times what I could have got a few months ago. I was amazed how much I cleaned out of my media collection when I decided to keep stuff I would actually want to watch or listen to again.

Re:Check out the SheevaPlug

[cynyr]

can i stream blu-ray rips off that? how about to 2-3 clients at once?

We will just not even address trying to use it for /home or / on thin clients. for storing pictures and music for your single laptop sure...

Delete stuff!

[Snaller]

Face it, you don't need to keep everything you ever thought about watching again - you are not going to do it anyway. Just delete some files ;)

Re:Thoughts from my home storage server experience

[CAIMLAS]

These are AOC SASLP-MV8 cards. They've got 2 SAS ports with which you can use SATA mini-SAS breakout cables (SFF-8087 - pretty standard faire) to get 8 SATA ports.

I've historically had bad experiences with SI cards, so I've steered clear.

I later picked up some LSI 1068e cards - actually, IBM BR10i cards, which are labeled SAS3082E-R. They work with everything. After some initial problems using the installed firmware, I flashed them with the IT (initiator) firmware, removing the "-R" functionality, so I can now use them with ZFS and md without issue. They're PCIe x4, and only cost around $120-$150 each. (They'll also give out some pretty good performance, FWIW.)

Which SI cards are you using, specifically? It'd be curious to have some ultra-cheap controllers for ZFS... not that $150 isn't already pretty damn lean.

Re:Mac mini or apple Tv

[RocketRabbit]

You can simply use NFS.

SnapRAID

[steverweber]

SnapRAID is an interesting opensource project that might hold a solution for this users RAID needs. Its kinda like unRAID, it's not RAID as we know it, but does have good recovery on data loss and is rather 'simple' to use. I think it's worth a look if you had a bad experiences with RAID5 in the past :( 'me'

Re:SnapRAID

[steverweber]

I feel the xmass drink, so im not holding back... Interesting how people reply and have not talked about SnapRAID (sorry if that changed).... Did they see that the opensource tool works for both linux and windows? did they notice that it saves power because drives power down /and that actually adds up after a year/? did they notice that it don't matter what file system they use or that they can disconnect the drive and plug into another system and read without extra crap? humm did they care or understand that sometimes the simple cheap solution is sometimes the best for home media centres? Sorry but this project should have a slashdot front page on its own... perhaps in due time. /kinda disappointed in the replies I got on my previous post/ wtf :P

Re:Endless Storage Expansion

[Master of Transhuman]

Just to comment on iSCSI a hair.

I have a client for which I set up an openFIler iSCSI box. The box connects to an unmanaged dumb GigE switch. Four iMacs connect to the same switch. Client uses FinalCut to access video assets on two MicroNet 4TB external storage enclosures. Each of the four iMacs accesses two drives in the enclosures configured as a 1.8TB volume.

Been running fine for, I dunno, two years? Box blew a power supply this past month, I grabbed a second box of similar make and model, reinstalled openFiler, attached the enclosures, ran a script someone on the Net created to reload the volumes, and reconfigured the iSCSI. Back in business in a couple hours..

openFiler is great, utterly reliable, In this case, running over the client's regular network has been no problem, since the path is box->switch->clients all on the same switch and his network doesn't really move that much stuff around in normal use since the twenty machines are mostly production machines.

I'm thinking of setting up an openFiler based backup server for the video department - in that case, I'll use a separate GigE network based on separate NICs and a separate GigE switch - although I'm not sure what I'll do about the iMacs (GigE USB, I guess, if there are drivers - someone on the Net has done it). The rest of the department is tower Windows PCs.

Re:Check out the SheevaPlug

[steverweber]

can i stream blu-ray rips off that? how about to 2-3 clients at once?

We will just not even address trying to use it for /home or / on thin clients. for storing pictures and music for your single laptop sure...</p></quote>

thats kinda crazy to be streaming 3 blu-ray videos 18GB? to clients in ~1.3hours in a home media centre.. but with a 1gbps lan you should be fine? perhaps the extra overhead is more then i would expect? Please fill me in. (18 GB) / (1.3 hours) = 31.5076923 Mbps... extra overhead... ?

My setup

[RobbieCrash]

I used to run Ubuntu Server with with one mdadm RAID5 array and one ZFS array using ZFS Native pool in RAIDZ1 as my file server. But I started running out of space in the RAID5 array, and was having massive problems with the ZFS pool under high IO (complete system lockups, deadlocks while flushing the write cache which killed SMB and NFS); 5Kb/sec writes and at most 5MB/sec reads after an initial 5-60 second _normal_ performance period.

I moved to OpenIndiana because I really like the long term and extensible benefits that ZFS offers, and the problems I was having seemed to only be happening to Linux users.

ZFS's built in, block-level data de-duplication means significant savings on episodic content. I'm currently running at 4% de-duplicated data. While 4% savings may not sound like much, I've got 4.5TB of TV shows that I'm storing in x264, and that yields a savings of around 100GB. I know that until recently, storage was dirt cheap, but even still, 100GB is nothing to scoff at. I've debated enabling the built-in compression, but I don't think I want to take the resource hit as my wife has recently started streaming things to her laptop while myself or our roommate are watching things on our respective computers.

Additionally, the ease of expanding and repairing my ZFS pool has made replacing hardware less time consuming, and has significantly lowered the performance hit while repairing the 'array'. Replacing a hard drive was dead simple with ZFS, and not nearly as nerve racking as the times I had to do so with my MDADM array. Additionally, I was still able to pull ~30MB/sec off the drives over SMB shares while the array was rebuilding. Writes also were similar over the network. Raw internal performance saw a significant hit, but moving things between ZFS file systems isn't something that needs to be done while rebuilding; using the media content is for my household.

Everything is running over Gigabit on cat5e with one Linksys 610N running DDWRT acting as a gigabit switch in my 'server room' (an uninsulated sunroom that's too hot to use in the summer and too cold to use in the winter), one crappy D-Link 5 port Gb switch , one Linksys e4200 acting as an AP and main switch, and an old Linksys WRT54GL running DD-WRT as my router and firewall. Recabling to Cat6 is extra expense for no practical gain. You're going to need it when you upgrade to 10GigE, but that's a few years away. By the time you start using 10GigE, Cat6 will be as cheap as 5E is now, and you still may not even see significant need to move to it.

Hardware wise, I'm running a system based on this motherboard, with 8GB RAM, this NIC, and this SATA controller, with a 80+Platinum certified power supply that I can't find a link to right now.

Solaris is installed on a Intel 320 80GB SSD, with 8GB dedicated as log space for my ZFS pool, and 40GB dedicated for cache space. I have one ZFS pool made up of two RAIDZ1 arrays. The first array is made up of 4 2TB WD Caviar Black (WD2001FASS) drives, and is attached to the SATA ports on my motherboard. The second array is made of 4 1TB WD Green drives attached to the LSI card.

Internal file moves average out at around 250MB sec for anything under 2GB, 800MB/sec for files larger than between 2GB and 7GB, and about 400MB/sec for anything over 7GB. Network writes are about 80MB/sec between servers, and about 40 max from elsewhere. which I attribute to the D-Link. Reads are also around 80MB/sec. I've been able to run 4 simultaneous 1080P streams without anyone complaining about stuttering, or excessive buffering at the start.

The system idles at around 40Watts, and under load pulls about 100. These numbers may be way off, because I honestly have no idea about how electricity

Here's my configuration

[LostMyBeaver]

My home server is a machine which I've been running for the past 20 years in one form or another. It's never been completely rebuilt, though sometimes I have come close as I've replaced everything but the case itself. But I like to think of it as the ever evolving server.

Currently, it is an entirely passively cooled system and with the exception of the drives making clicking sounds on occasion is utterly silent.

1) Big roomy case. I'm using an I&S 4U rack mount case which has room for 8 vertically mounted 5.25" hard drives. I have a homegrown rack made out of kitchen counter top material and rack rails and a few screws mounted on rubber coated wheels as well. This makes it even quieter.
2) I lied about the 100% passively cooled. The power supply is not passively cooled, but by using a 1000w power supply for a system which uses about 100watts at peak load, the fan on the power supply serves as the CPU fan as well. I have to put my head up to the machine to hear it though. Search and you will find one... they're easy to come by.
3) 8x 2TB Seagate 5900RPM drives. They're cheap (or were before the "shortage") though I'd use 3TB or hold off for 4TB drives now.
4) Highpoint Rocket RAID 2720. It has 8 ports on it and has proven its worth already (I'm using the previous model, but this one is better) since I run RAID 5 and have survived two hard drive failures in the past year. The 2720 is worth its weight in gold because it has full support for SAS expanders, so this controller is actually useful for hooking up 24 drives without fans by using the EJ240 SAS expander in another case.
5) Using super cheap $1 each 3.5" to 5.25" brackets, I mount the 8 hard drives vertically in the chassis. This leaves A LOT of room to keep them cool passively. Perfect!

These are the easy parts... the trickier part is getting the right motherboard for the software. I personally run Windows Home Server 2011 which is perfect for my needs. I bought a copy from an online vendor for $35. It has DLNA, has something like Time Machine for Windows computers and with some other downloads, I have NFS and AFS+ support. I added a licensed copy of Alcohol 120% which allows me to mount CD/DVD images on other computers as iSCSI and it even emulates the copy protection schemes, so when I buy a game DVD, I can use it from any computer in the house without having to use NO-DVD cracks. I also run Air Video Server which allows me to play pretty much any file from the server to iDevices without reencoding.

So that's the point.. being able to encode multiple streams for streaming in real time and still being passively cooled. At the moment, I'm using a power hogging Core 2 Quad Q8400 CPU.... I'm not pleased with the power consumption and I have been running a few trial runs to see if a 45W Core i5 2500T would work for my needs. Using Intel QuickSync, it is proving to be a perfect solution as the CPU, even when re-encoding videos uses almost no power. Also, but moving the video card into the CPU, it eliminates most of the remaining power foot print of the system.

Then the last remaining item was purchasing memory which is low power as well. I tried 10 different sets of DIMMS and eventually found that power usage on RAM could vary as much as 10 watts for 8gigs.

I log power consumption of the machine 24/7 and find that the machine uses 25 watts when it is idling and peaks at 100 watts in bursts. Overall, the average power consumption of the machine is 37 watts.... less than the cable set top box provided to me by Altibox in Norway. (with the core i5... it's closer to 65watts with the Q8400 and the graphics in the chipset).

Oh... ONE LAST THING... the boot drive. I don't boot from the RAID. I boot from an SSD. I bought a cheap 60GB SSD for $60 from a "fire sale" which is slow... only 120MB/sec.

I'm sure that using Myth server or something else on Linux, this configuration would work well for you too. It's a little pricey... but it gives me 14TB of formatted space. If low power 4TB drives come out sometime soon... I may replace the 8 drives with 4 of those and then just add one as needed... I can afford losing the last 2TB since I'm not using all 14TB just yet.

Re:Thoughts from my home storage server experience

[jafo]

Nono, the cards I'm using have 8 discrete SATA ports on them. See the photo on the URL that I pasted in my original comment.

The cards I'm using are all fairly old at this point, they weren't exactly the latest stuff in 2008 when I put together my storage server. They had a really good reputation then though. The SIlicon Image controllers I really don't remember, but the model number 3114 sticks in my head, they were 4 port PCI cards.

Backblaze

[SkipF]

Backblaze, an online storage company has posted the specs for their rack mounted box. http://techcrunch.com/2011/07/20/backblaze-presents-their-bare-bones-7348-135tb-storage-pod-for-backup-on-the-cheap/

TV output.

[antdude]

One thing that pisses me off is the latest video drivers don't work with fullscreen overlays on TVs. NVIDIA and pulled this useful fullscreen video feature from their latest video drivers and Windows. They say it is for copy protection/DRM (lame!). ATI can do it but for me, I have to use DVI and CRT (no VGA since I use an old KVM from Y2K) only. :(

Qnap

[Anonymous Coward]

Check out linux based Qnap NAS devices. They have big model range from home to corp. Very power friendly and nice GUI.

Re:I was saying maybe a raid card but non raid car

[nabsltd]

You do know, any ROM flash is just software raid right? Even SAS controllers don't have hardware RAID unless you buy a real raid card for $$$. Real raid cards have write back memory and a BBU.

I never claimed the motherboard had hardware RAID...I just said it had 8 ports of SAS on-board. As an add-in card, that will set you back around $100, so I was trying to show how a true server motherboard is a better deal than a cheap motherboard that you have to add everything to. Also, the SAS controller has 8x PCIe lanes dedicated to it, so that you can even use SSDs and get their full speed.

I do use Linux software RAID, with 5 drives on the on-board SAS, and 5 on my add-in card (which is full battery-backed hardware RAID, but I'm using it in JBOD mode, 'cause it was handy). The RAID-10 splits each mirror set across the two controllers. I tried RAID-5, but it only gave me about 200MB/sec write speeds, and I get nearly 350MB/sec with RAID-10.

Re:I was saying maybe a raid card but non raid car

[nabsltd]

Not that I really believe the mobo comes with a surface-mounted i7 but even if that's a "suggested" processor, it's stupid, since this is a job for an i3 or better yet, Celeron (or Sempron or Athlon II).

For a server, you want lots of threads, ECC memory and lots of PCIe connectivity, which this has. I use a W3520 processsor, but disable hyperthreading so I have 4 cores.

Then there's all the SAS ports, which is theoretically very cool until he goes to buy drives and .. well, just check the low capacities and the prices.

So, buy SATA, which every SAS controller supports.

Ooh, and it can take 24GB of RAM, as if you need more than the minimum you can buy these days. 2GB will be overkill.

With 8GB of RAM, Linux is using nearly all of it as disk cache. This gives me in the neighborhood of 900MB/sec reads for cached data for the VMs (over 10Gbps Ethernet). Since I don't cache the media (movies, music, etc.), but only the OS and other small data, this gives huge performance benefits for stuff that matters.

He's going to have you build a really bitchin' machine but that's not what you want [to pay for]. DO NOT buy a "server" board. Most servers don't need "server" boards.

If you want to run something 24/7, you need to pay for quality. Since the OP also wants lots of connectivity, he's better off paying $200-300 for a better motherboard instead of paying $100 for a cheap one that needs $100-200 in extra cards. Also, if his needs grow, he won't need to replace the motherboard...he can just add a card or three.

Re:Xbox Media Server

[Killjoy_NL]

Glad you liked it :D

Keep it simple

[DedTV]

The setup I use is made for simplicity. I wanted something that worked, not a new hobby. And I have built them for friends and family and wanted something anyone with just a little computer skill to be able to manage themselves so I wouldn't be getting called to fix things every 10 minutes.

I use one of the Norco 20 or 24 bay Server Cases ($300-400 on Newegg) for the server. It's a rackmountable case but works just fine sitting on a table or a sturdy closet shelf. After swapping out the Delta fans it comes with, it's fairly quiet. I have mine sitting on a small table in my home office and it doesn't bother me at all.

The hardware inside can be whatever you want and can be added to and upgraded over time. I started with a Supermicro 8 port SATA card and 6 1TB HDs in a JBOD setup on a Core 2 server mobo (for a couple PCI-X slots) I picked up on Ebay. Over the course of 3 or 4 years it's now on an Intel I5 system with 20 2TB in Raid 5 using a fairly cheap Areca hardware raid card and an HP SAS Expander Card. I added the HDs over time, buying one every month or so at first and then buying 2 or 3 at a time once they started getting cheap. Thankfully I finished and have plenty of warranty left in case of a failure now that they're not.

I also tried a bunch of NAS OSes, Linux distros, and WHS but they all had something that annoyed me when I tried to use them as a media server OS. For something as simple as serving media files for the home, a simple OS like Windows is perfect. I just threw on a copy of Windows Server 2k8 from my Technet sub (Win 7 would be fine as well. 2k8 is just a bit easier to run headless) and added Sickbeard, Coach Potato and some ripping and conversion tools.

To play media, I use XBMC. It'll play anything, including ISOs so I don't have to rip and convert my movies and it's very easy to manage.
I used to use old computers and build HTPCs out of them but now I use things like Acer Revo, Zotac Zbox or Apple TVs. Whatever's cheapest that's small and can play HD video. I load XBMC on them and attach them to the back of the TV with Velcro, run an IR receiver (if the mini-PC doesn't have one, I buy a cheap one on Ebay) and I'm good to go.

The hardest part of my setup was that house wasn't wired with Ethernet and I don't have the skills or confidence to start drilling holes to run it myself. I had an electrician come out and run conduit to each room in the house where I'd ever want video and then ran Cat 6 through that. With everything being wired through a GB switch, everyone in the house (up to 4 TVs going at once) can be watching something off the server and we rarely have any stuttering and I spend very little time administrating it.

Re:Mac mini or apple Tv

[dargaud]

I had same issue. Disable smart in Bios and kill or tweak smartd in linux.

If you disable SMART in BIOS, can you still use SMART in the OS ?

some ideas for you

[cas2000]

you can buy a 20 or 24 bay case for around $300-$400 US, e.g. Norco RPC-4020 or RPC-4224. Takes up to a full size EEB 12"x13" motherboard and 20 or 24 3.5" hot-swap SAS/SATA drives. Can take a standard power supply or there are redundant dual power supplies available.

http://www.norcotek.com/item_detail.php?categoryid=1&modelno=RPC-4020
http://www.norcotek.com/item_detail.php?categoryid=1&modelno=RPC-4224

The 24 port version has a nice option to replace the internal fan bracket (which supports 4 x 80mm fans) with a bracket that supports 3 x 120mm fans. Much quieter for a home environment. Dunno if the RPC-4020 has a similar option. You *WILL* want to replace all of the supplied fans with third-party silent fans. http://www.silentpcreview.com/ is a good place to start researching this.

Even if you're only planning to have 10 or less drives right now, the extra bays are useful if/when you need to replace or upgrade existing drives. You won't have to juggle drives in and out of bays just to replace them. or have a drive hanging outside the case for a few hours while the data is copied.

For extra SATA ports, there are several models of LSI 9211 and similar HBA adaptors providing SAS/SATA 6Gpbs, PCI-e 8x slot. RRP is around $350 for 8 port models but you can find them cheaper on ebay, and several manufacturers (e.g. the IBM M1015) have significantly cheaper rebadged models. A SAS card allows you to use either or both SAS and SATA drives, and also allows you to use SAS expanders (to attach more drives to the one card - SATA has something similar called "port multipliers" but it's a crappy substitute only good for destroying your data). Unless you don't have enough PCI-e 8x slots in your m/b, though, you're better off just buying more 8 port cards.

They're just "dumb" HBAs offering only RAID-0, RAID-1, and JBOD....but that's exactly what you want for software raid or btrfs or ZFS so why pay extra for RAID-5 in the card that you're never going to use.

The LSI 1068 based cards are even cheaper, but they only support SAS/SATA 3Gbps. Doesn't matter much for current hard disks, but you'll need a few 6Gbps ports on the motherboard if you want to use SSD drives (e.g. for caching.)

here's a good starting point: http://blog.zorinaq.com/?e=10

see also http://forums.servethehome.com/showthread.php?19-LSI-RAID-Controller-HBA-Equivalency-Mapping

For the file system, I very strongly recommend ZFS On Linux (the native kernel implementation, not the ZFS-Fuse module). http://zfsonlinux.org/ - gives you raid-like features, disk/volume management, compression, de-duping, snapshots, ssd caching and more. all data is checksummed too so it can detect errors (and automatically repair them from redundant info on the RAID1/5-like volumes).

The Ubuntu PPA compiles easily on debian (you only have to change one dependancy from zfs-grub to grub in the debian/control file) - it's about 10 minutes work, and most of that is waiting for the packages to compile.

ZFS will give you software-raid like capabilities - superior equivs to RAID-0, RAID-1, and RAID-5/6 and combinations of them, plus multiple optional hot and cold spares. "superior" because the redundancy is on the file/data level, not at the block level, and each block of each file is checksummed. Plus you can use one or more fast devices like an SSD for automatic read caching of frequently access data (ZFS cache or L2ARC), and for a write-intent log (ZFS ZIL) for buffering random-writes to an SSD before writing them to the main drives. This ZIL eliminates the final advantage that hardware raid cards had

Re:Mac mini or apple Tv

[Oxyde]

It's a bit of a pain to remux mkvs to mp4s but it works

What software do you use to remux?

Best Kit for Home Media Server

[KC117MX]

There are several ways that you could increase the storage on your system. One would be to increase the size of each of your physical disks and keep your existing setup. Another would be to add SATA cards with eSATA external plugs and add additional drives to an external eSATA enclosure. My concern with external enclosures, however, is heat dissipation. Heat is often a destroyer of drives, but there are external enclosures available that address that problem. You could replace the case where you could add additional drives, but eventually you would be back in the same situation that you are in now. However, a combination of these options would not only allow you to currently increase the size of your storage but also allow for you to have room to grow in the future. That would not be the most cost-effective approach for short term growth but would be most effective for long term growth. Replace the case with a case that has more expansion bays for HDDs, add eSATA cards, upgrade some of your drives, and put the old drives in external eSATA enclosures. If you wanted to go to a rack system, then you would have substantially more room for growth, a specatucular setup, but would not be near as cost-effective. As for your HTPC software, I currently use XBMC in a HTPC case using an old PC with a core2 duo CPU and several gigs of RAM booting from a 4GB usb drive. XBMC covers everything that I need and streams my movies(h.264)/music(mp3)/pictures without an issue. I even bought some fancy remote to work with it, although my family prefer a wireless mouse over the air-mouse. My server uses an Antec 180 case with a AMS SATA Backplane Module modded to fit into the 5.25" bays. I still have room for a Blue Ray burner and I can fit 11 drives in the case, for a current total of 31 GB inside of one case. I use five 120mm and one 80mm SilenX fans in the case for quiet/reliability/performance. My two cents. Hope it helps.