Creating Large, Safe, and Cheap Network Attached Servers?

davco9200 asks: "I am looking to create a large data server for all my digital media files. The usage is the 'pro-user' category, to use the media from multiple stations in my house and at work. I value space (150+ gb would be nice), accessible from multiple platforms (Win, Mac), but perhaps most of all, some security (e.g. RAID 0 doesn't cut it). Total write or read access isn't that high of a priority. I have looked at things like the Snap 4100 that offer 160 gb or 300 gb and good raid options but the price seems high ($2,700 and $4,500 respectively). Has anyone had any experience making their own low-end NAS? Has anyone looked at the Adaptec IDE RAID Controller? This seems like a reasonable way of getting data parity so if one drive goes down your entire collection isn't lost. I figured Slashdot readers would have some good solutions. Information on specific cases, drives, and other pertinent facts would be helpful."

Comments

Well...

[cmowire]

Well, if you don't care too much about performance, don't even bother with a 'real' IDE RAID card, just do software RAID. The reason why people use the hardware RAID cards like the higher end IDE RAID cards and the SCSI RAID cards is because it's faster.

So you can cut the cost of a good IDE raid card and just put an extra IDE controller card in your psuedo-NAS box so you can have 8 drives. Or you can put that off until later.

You should also consider getting a DAT drive and a bunch of DAT tapes, to back things up, just in case something massively bad happens to your system. RAID 5 is not perfect, and if your system catches on fire because of too much dust in the power supply, it'll be helpful.

Re:Well...

[JabberWokky]

I wouldn't let DAT tapes anywhere near backup of stuff I actually care about for several reasons, but mostly, capacity and speed

Bzzzt... we're working under a different set of rules here. I'm in the process of doing this exact same thing, and I'm using DAT for backup because they are perfect for what I want - long term, cheap, backup once and never again storage media.

Think of this way - who *cares* if it's a small box of tapes? Who *cares* if it takes 48 hours to back it up? I'm converting my entire CD library to MP3, and my entire videotape/DVD library to DivX ;-). As I get about 30-35 gigs of encoded data done (which takes quite awhile to encode), I back it up onto a single DAT tape. That's 50 movies, or 3500 songs... which will free up two shelves of videotape (or really one shelf, since I have to double shelve tapes), or a chunk of one of my 500 disk CD-changers, and turning two shelves of VHS into a very very small, durable tape is very very nice.

As you can probably guess by that last sentence, I have lots of media... tons of CDs and video. I have this fantasy that I get raided for "piracy" because I use MP3 and DivX ;-), it hits the news bit time, and I walk into court with dozens of boxes full of original videotapes, DVDs and CDs. Hehehehe.

My prototype system is a 450Mhz Pentium II with a Promise IDE RAID, with one Maxtor 80 gig hooked up, 64M RAM (I had two 20 gig hooked up to test the RAID earlier). I'm looking at upping the processor because encoding just takes too long, the memory seems fine for single playback or encoding, and the controller seems fine for 4 drives, but I want 8, probably Maxtor 80 gigs, as they are cheap.

My biggest problem is the initial video grab or analog sources. I haven't found a really good capture card, and I want one that runs under Linux, preferably with open source command line utilities so I can automate the process. As it is, I can convert DVDs just fine, and I have an All-in-Wonder Pro sitting on my desk that I want to test (since I have it, I might as well try it). I've seen some absolutly incredibly encoded porn out there - very clearly grabbed by an amatuer, and from a video (you can see the artifacts), but the clarity from an NTSC signal is incredible. I'm really curious as to what equipment is available for the prosumer for video capture, preferably Linux friendly.

Alas, as I just laid myself off from my company (at least for a few months until we start making better revenue), it dosen't look like I'll finish my media server for another several months. Maybe then I'll get 120 gig drives.

--
Evan

Re:Well...

[JabberWokky]

AIT-2 whacks your DAT upside the head with room to spare.

Very good... and if I were a company with money to spend, needed to save time, or if I was a millionare masturbating with my system stats, I would get one. As it is, I've had the DAT drive for years, the media is dirt cheap, and HP's units are very reliable.

What part of "define your operating needs" didn't you understand in all this? We're looking at a homebuilt system out of cheap and available tech. I'm willing to trade time for money (as it is, a 450Mhz PII is perfectly fine, even if it's a "drop DVD in, and the next day, you've got a DivX ;-) file" situation or "drop a CD in, and do other stuff while it takes its merry time to encode"). While a more sexy bulk storage solution might net me space and time savings, I don't *care* if I've got a tape briefcase full of DATs versus one unit. Hell, for that matter - if that one media fails, I've potentially lost it all. If a DAT fails, I lost just part of the collection, and reencode just those. Speed of backup does not matter. Once backed up once, I don't ever do it again (although when it's all done, I'll probably make another backup of home videos and a few others and toss 'em to my relatives for safekeeping - cheapman's "off-site backup storage").

If I wanted to do it sexy, I wouldn't have tossed two 32Meg dimms in there, used a dirt cheap Promise RAID controller, or encoded to DivX ;-). As it is, the VHS tapes will go nicely to DivX, and I'll probably keep the DVDs on hand because DVD to DivX ;-) is a noticable loss in Audio and Video quality on my HDTV projector with a Kenwood full fiber 7.1 sound system (even though I run video though a analog video line to smooth DVD artifacts - we call the direct digital to HTDV on a seven foot tall screen the "Artifact Detector mode"). Still, for light fare like Urusei Yatsura, where the original wasn't high quality, I'll just watch the DivX ;-). For things like Apocolypse Now, Clockwork Orange, or The Wall, I want the best damn audio and video, and I'll go to the DVD for that (or the laserdisc). I'm willing to lose a bit of audio quality on CDs in exchange for flexibility, but I still encode at 256k. This is a hobby project born out of spare parts and fulled by the availability of dirt cheap drives and RAID controllers... and a desire to reclaim two damn rooms worth of videos and CDs, and something like 15u of space on my primary A/V rack (CD changers).

--
Evan

Re:Well...

[JabberWokky]

Sounds like you are already a masturbating millionare, or a snot nosed 14 year old. Personally, I think you are just a worthless troll.

You'll learn when you get married to or live with a female geek, a combined income allows you to buy the nicer things in life... until you have kids (which I've avoided to date).

The only really expensive item in my entire setup is the projector, which was $5800 refurbished. And I got that primarily because I host Rocky Horror rehearsals and the South Florida Anime and Manga club meetings (SFAM), plus, while I don't watch TV, I really like movies.

--
Evan

Solution

[1101z]

I am working on a similar project but in the >1TB range same thing applys to http://staff.sdsc.edu/its/terafile/ On that page they have a link to another page with stuff about ide-raid. >http://www.research.att.com/~gjm/linux/ide-rai d.html The 3ware cards are the way to go as they do raid 0,1,5 in hardware, and support things like hot swap and hot spare. I priced out a system that was just over 1TB of raid 5 for around $5,000 while the prebuilt stuff is $20,000.

Used parts make it cheap...

[mike_the_kid]

One great place to look for an old tower is your local Computer Goodwill or used bulletin board. Get an old box with a 300W power supply, take out the motherboard, start stacking SCSI drives in a daisy chain, and then bring it back to a SCSI controller on your main box.

Obviously SCSI is more expensive than IDE, but you get a little bit more. Just food for thought.

Windows 2000 Software RAID

[man_ls]

Tom's Hardware has an article on their main page about software RAID under Windows 2000. Apparently it supports striping and spanning, and unlimited drives as long as you have enough controllers for them. The boot HDD has to be standalone but the others can be RAID-linked. The best thing is that it supports multiple interfaces. You can have 3 IDE drives and a SCSI drive and they'll all RAID together.

I don't know if an MS product is what you'd want to use, but it's out there.

J.W. Koebel

Re:Windows 2000 Software RAID

[zulux]

The trouble with Windows 2000 software RAID is that you boot disk can't be a part of the RAID array. I'd recommend the Windows 2000 software raid over the Promise ATA RAID cards - they are software too, they just hook themselves to the BIOS. The Promise ATA solution has been flaky in the two instalations I have used it - I had to use IBM's DFT to make the two IBM drives communicate to the Promise FastTrack controller in UDMA 2 mode - otherwise they would get corrupted. I've had very good luck with 3Ware's hardware products, highly recomended!

use those 3ware controllers

[mbyte]

I've build a box just for that requirement a few months ago using a 3ware IDE controller, 4x100gb drives in raid 5 mode and linux :) gives a nice 300 gb of space.

I think the overall costs were around 2000 Euro ..
(controller, drive bays, BIG case, etc ..)

Homebrew Firewire NAS

[Vito]

One way to go might be an inexpensive, but not underpowered PC, with a PCI Firewir-- er, IEEE-1394 card.

Buy a bunch of cheap, identical IDE HDs, and put them in IEEE-1394 cases (~$150/ea.). Compile yourself some bleeding-edge Linux-1394 support, plug in your HDs, run XFS as the filesystem, and use software RAID. Because you said this is just for storage and media access, you probably don't need the currently limited FireWire hot-plug support and possibly still currently limited RAID hot-swap support.

For more on software RAID, IBM has a nice two-part article (1, 2) on it.

Homebrew Snapserver 4100

[il_seba]

This is my receipe for an "homebrew" Snap4100

1) Get:
- 1U 4bays rack mountable chassis from Sliger Designs
- 3WARE 6410 Escalade IDE controller (Choice of 0/1/0+1/5 Raid) on a 90 PCI riser card
- 4 x 75/100GB ATA100 drives (maybe DiamondMax)
- MicroATX mainboard with NIC and Video integrated on board (invest in RAM not in processing power - 750/850MHZ should be more than sufficient)
- Minimum Linux/*BSD OS booting from a read-only 16 to 64MB flash IDE device, loading kernel and a customised Ramdisk root filesystem, mounting Raid devices in R/W mode, starting SAMBA (and/or Netatalk).
A good starting point is Linux Bootdisk HOWTO

2) Choose 0+1 Raid and you get quick and completely redundant 150/200GB storage that can survive the full failure of one disk.

3) Want remote grafical managment from a standard web browser? Go for Webmin or SWAT.

Don't be *too* cheap!

[coyote-san]

Don't be too cheap here. Power supplies do go out (or catch fire because of dust or pet hair that blew into them) and even a "good" unit might have problem maintaining its rated capacity after a number of years.

Keep the chassis, but replace the power supply. Besides, since this is a server this sounds like a good place to use one of those units with a built-in UPS, even if box is hooked up to an external UPS. (I've had them fail because I unwittingly overloaded them, because of poor designs that allowed me to accidently turn off the UPS protection but not the power, etc.)

Pretty easy

[Sloppy]

Has anyone had any experience making their own low-end NAS?

Yeah, probably about half the people on Slashdot. ;-)

There's really nothing to it. Get a x86 PeeCee, 4 cheapo 80GB ATA drives, and Linux or xBSD. Put a bootable+usablesystem partition on each drive (since you never know which drive is going to fail first) and use the rest of each drive as a slice for a RAID5. You'll have a about 220-240 Gig of storage that can survive media failure. You can do it for somewhere in the $1500-$2000 range.

For larger non-media failures, you're still screwed, though. Backup technology just hasn't kept up. :( Unless you have Big Money to spend on this (and maybe even then), you will end up using many tapes for a single backup. That fact will tend to influence you toward not backing up at all, unless you are unusually disciplined.

For the exporting to the other machines, the stuff you need should come with just about any Linux distro. I use NFS (for Unix and Amiga) and Appletalk (for Mac). I suppose Samba would work for wintel boxes.

Re:why NAS

[SlashGeek]

What you missed was the urinal this morning when your little dick pissed on your nuts.