Slashdot Mirror
Best Solutions For Massive Home Hard Drive Storage?
i_ate_god writes "I download a lot of 720/1080p videos, and I also produce a lot of raw uncompressed video. I have run out of slots to put in hard drives across two computers. I need (read: want) access to my files at all times (over a network is fine), especially since I maintain a library of what I've got on the TV computer. I don't want to have swappable USB drives, I want all hard drives available all the time on my network. I'm assuming that, since it's on a network, I won't need 16,000 RPM drives and thus I'm hoping a solution exists that can be moderately quiet and/or hidden away somewhere and still keep somewhat cool. So Slashdot, what have you done?"
How much data constitutes "massive"?
Do something like this. Put it in a case / box / cabinet of your own design since you don't need the rackmount capability.
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
I really like the DroboPro. It's pricey but you can install up to 8 drives, upgrade drives as necessary to increase storage, and it's quiet/fast/low-power. You'll still need a file server since it's a SAN device, not a NAS device, but I'm happy with my 16TB configuration.
DLink - DNS-323 with two WD 1 TB Green Drives. Quiet, works out of the box and is also Linux hackable if you feel the need.
Enjoy!
"I say we take off, nuke the site from orbit. It's the only way to be sure."
Personally I'm using a Synology solution at the moment, for my NAS. They offer a relatively low cost feature rich hardware, with low power and depending on the HDD's you use, lower power consumption than that of a always on PC.I've been thinking about later on upgrading, since a general rule of thumb, you can never have to much storage. For HD BluRay images I would recommend making sure the network isn't the bottleneck and use gigabit ethernet, as I'm finding on my aging 10/100 switches it's not cutting it. xvid's and MP3 streaming it seems to be fine.
So Slashdot, what have you done?
Why? What have you heard??
My personal storage solution consists of a 4U rack case with a computer with a c2d CPU, gig-E NIC, a few gigs of ram, a bunch of 7200 RPM disks and FreeBSD on the system disk (I also have the system disk mirrored just in case). All the storage disks are then pooled using RAIDZ. Pretty simple yet powerful. Just don't expect too much in the way of performance.
Greylisting is to SMTP as NAT is to IPv4
..., are the movies you download compressed at all? You say you run out of slots, how big are the drives you're putting in the slots? Personally, I let Netflix do the storing for me. I have a few TB's but never come close to filling it up.
I used to work for ABC news and we never kept archive footage always accessible like you want. If we wanted something that was really old we'd have to dig it off a tape, an unplugged hard drive or powered off computer, or we'd have to find another news agency that had the footage and grab it off of a satellite feed. And this was a 24/7 TV news station responsible for national news programming where we would be tracking stories for years. If we didn't need a system where everything was instantly accessible then you needing it on an individual level might be overkill in my opinion.
I have over 30TB of music, movies, and raw video footage on my home computers and I just keep everything on separate external hard drives. I label the drives, back them up twice each, and then keep an index in a .txt file that is easy to search through. So if I want a 1080p backup copy of Blade Runner I search 'Blade Runner' in the .txt file and I see it's on drive 'A' and then I plug in drive 'A' and dump the movie on my computer. I also keep an external drive that has backups of every TV show I own on DVD. So if I want to watch The Wire then I plug in the external drive labeled 'TV' and have at it.
lets say 5 GB per movie - I have 200 in my downloaded collection and that is just movies people have recommended. Add in some tv shows and probably some music and lets exaggerate a little. So 5 TB would be a great start.
Buy a mobo with a lot of sata slots (read: as many as possible). Then load it full of 1.5 TB drives. run an outlet and a cat5e (or cat6) to the towel closet and hook it up on the top shelf. problem solved.
Don't worry, we'll be right over and take care of everything. You'll never have to worry about it again.
MPAA
P.S. My sister, Riaa wants to know if you're into MP3s
SATA port multipliers - 5 to 1 for about $50 + 5 2 TB gives you 10 TB off 1 SATA port.
Not the cheapest, nor the most expensive but heres my solution:
Thecus 4100 Pro
4 x 1.5 TB Hard Drives in RAID for 4.5 TB of Storage
The Thecus 4100 pro has lots of capabilities that I just haven't taken the time to use/learn yet. But it's been very effective for me so far. This is my first NAS build. It's great to be able to access media all over the house and now serves as my primary storage rather than the 4 x 1 TB's I had in my desktop and other externals that I had spread out.
"What is there a tank on the boat? WHY IS THERE A TANK ON THE BOAT?!?" L4D2
I personally set up a downloadserver that also functions as a media server to stream the content to other devices. I put in a couple 5400 RPM 1,5 TB drives, they use less power, generate less noise and heat than a regular 7200RPM drive but since you're not running any applications of them, you won't really notice the difference in performance. Prices have gone down a bit so the sweet spot for $/GB might be at the 2TB mark now. If you don't want to go for an entire computer, maybe a NAS solution would be best for you, with the same 5400RPM drives. A NAS will have less room for the disks if you really want *massive* amounts of storage, and also you usually must purchase one + the disks. The PC you can build from spare parts lying around. I personally put gentoo linux on mine, but you also don't exactly need top of the line equipment for a nice windows XP install. The NAS however will have outputs directly for your TV and will take up less room and power.
Still, the key is 5400 RPM + 1,5/2 TB.
/. is definitely the place to ask...
Do you have ESP?
Linux Software RAID5 has worked very well for me. Performance is decent (perfectly fine to play back and transfer 1080p video). I got one way back when 3x320GB was enormous and had a 1TB drive before they were remotely available.
Now I'm seriously considering 6x2TB for a 10TB RAID for my next server replacement. No need for an SSD for booting either, just set aside a tiny RAID1 partition (mirrored across all drives) for /boot and you're set. It boots and operates fast enough.
The one problem (as with any solution here) is that 10TB is nearly impossible to back up. Assume the data is lost in case of server hack/house fire, and back up that 50GB that's *really* important to removeable media offsite. I've got an external USB SATA drive desktop "plugin" that works well for larger file transfers. (And Hard Drives are now getting very close to DVDs in GB/$)
I'm sure someone will; pipe up about how SATA drives aren't stable enough for RAID. ZFS is the alternative, but I'm not sure what tools are available for Linux distributions.
Windows home server, 1TB 7200rpm main drive with seagate LP 5900rpm drives, lock it away and never have to think about it till you need to drop another drive in.
The reason for the fast main drive is that with WHS when you copy data to it, it stores it on the main drive first, then schedules it to be distributed out to the storage drives the next time a "storage balance" is done.
Works fairly well, its based off windows server 2003 at the moment, but if you can wait till the end of the year they have a server 2008r2 version coming out soonish.
...
More than that, you might not need even 7200 RPM drives. There are large capacity "green line" drives from some manufacturers, 5400 RPM, that might be perfectly enough.
I'm sure other posters will have much better recommendations as to how the overall setup should look like, but for whatever it's worth from me - stay away from consumer NAS solutions, they have usually quite small transfers (and I guess its important to you, with files being rather big). Large tower with plenty of space inside + Atom motherboard should be enough, otoh (as long as that Atom board has enough SATA ports...)
One that hath name thou can not otter
I download a lot porn, and I also record a lot of masturbating videos. I have filled two computers with porn already. I want access to my porn at all times, especially since I maintain a porn site. I don't want to have swappable USB drives, I want all my porn available all the time on my network. I'm assuming that, since it's on a network, I won't need 16,000 RPM drives and thus I'm hoping a solution exists that can be disguised or stashed away and not overheat. So Slashdot, what have you done?
I've tried a variety of approaches, but overall I've been happiest with just buying a NAS box.
I have a Synology DS209, and I've been very satisfied. It's a relatively cheap way to get 2 TB RAID 1 storage with really simple backup to an external USB drive. If you need more storage, you can buy NAS devices with more than just two bays.
You don't want swappable USB drives, when then give us more details. How much do you want to spend? $100, 1000, 10000? 1TB USB drives are under $100 each. You can pay that or up to $500 for a NAS in a can for 2TB. The next step up is a group level rack mount SAN/NAS but expect to pay about $500-1000 per TB. Other than that, you will be piecing something together and will undoubtedly be much more complicated or time consuming but no one here can give you a solution for that because you have not given enough details. Search Google for various LaCie NAS devices, they seem pretty cheap. If you are the only one accessing this shared storage, USB 1TB for $100 really is a good solution.
Here are some of my experiences (I've published some more information at http://www.tummy.com/Community/Articles/ultimatestorage2008/):
If you just want super cheap storage, and really don't care about occasionally losing it, just use a collection of simple USB enclosures or network storage devices with individual file-systems on them. This way, data loss may be isolated to a single device, but you get little in the way of error correction.
If you care about the integrity of the data, use ZFS with RAID-Z. RAID-Z checksums the data and can recover from drive corruption. While it's rare, I *HAVE* over the last 3 years seen a few occasions where it corrected errors that normal RAID-5 probably wouldn't have corrected. If you don't particularly care about detecting bit-rot, this is optional.
If you care about the data, make sure you have backups. Again, for video you have downloaded, this may not be that important. In my case, I store all my old photos, music (which I *CAN* re-rip, but don't want to ever have to), so I built two systems.
If any of this data is private, use crypto. For example, we store scanned copies of our records. If someone steals this computer, I don't want them to also walk away with all this private data. Again, maybe not a concern for the OP.
Just get a case that will hold enough drives for your needs, don't try to be fancy with external storage, USB enclosures, etc. External storage sounds like a good idea, but these additional connections are places that can fail. External SATA is a nice idea, but I've had tons of problems with them. The connectors easily come undone, and 2 out of 3 of the 5 drive enclosures I got (see above URL) have been nothing but problems. Luckily, the one on my main storage server has been absolutely no problem. Doing it over in the future, I'd be tempted by something like the Antec twelve hundred which can hold 9 drives internal or up to 20 via the "5 in 3" internal enclosures (or some combination of the two).
Don't worry so much about a quiet machine, it doesn't need to be anywhere that it matters. Mine is in the furnace room.
Of course, set up regular RAID array verify runs and array alerting so that you find out when the first drive in the array fails, not the second.
Sean
Set up a nice OpenSolaris box with ZFS and export it with Samba/NFS/iSCSI, etc.
You can tell how powerful someone is by the magnitude of the crime they can commit and be able to get away with.
You made no mention of a budget. I'd go with a Drobo - probably the DroboFS. http://www.drobo.com/products/drobo-fs.php
Informatus Technologicus
I suppose if you like fiddling and want to tweak, then building your own is fun and all but if you just want something that works, is most likely quieter and uses less power than one you build yourself, then I say a standalone NAS unit.
I have a QNAP which I love - Synlogy, D-Link, Thecus, Buffalo, etc etc there's a lot of choices out there in 1/2/4/8+++ drive bay sizes. They will typically have various RAIDing options, spiffy web management interfaces, etc that make 'em pretty plug and play.
Just make sure to get one with DLNA support if you want to do streaming to entertainment systems.
... you should totally check out Cali Lewis's promo video at http://drobo.com/resources/drobodemo.php She is soooooo cute. Plus I recommend the drobo as well with its cool "Beyond Raid" system that let's you just pull out the smallest disk in your array and plug in a new one. Anyone who's ever rebuild or updated a traditional RAID array knows what an improvement that is. Good Luck
Try The Black Dwarf technique: http://www.willudesign.com/BlackDwarfTop.html
...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
E.g. http://www.openafs.org/, http://www.gluster.org/
The Internet's nature is peer to peer - 20050301_cs_profs.pdf
I have two of these servers now. Each server can hold as many as 16 disks (possibly more actually as the programmer keeps bumping that up) with one disk reserved for parity. Data is NOT striped and parity is ONLY stored on the one drive. If a disk fails I lose no data, if two fail I lose two disks of data but nothing else. No hot spares or any other crap. If a disk isn't being used it goes to sleep and saves me heat and power. Disks can be ANY size but the parity disk must be as big or bigger than any of the data disks. Runs on a pretty decent selection of hardware although keeping the list of what works and what doesn't up to date is apparently tough since hardware changes so fast. It's Linux based but pay for play, yes he's followed the GPL. It's not super expensive and it boots from a USB drive to be web administered. I use full tower cases with SuperMicro 5n1 trays, 2gig of memory, Celeron CPU, power saving PSU, and supported mobo that have onboard video and GigE which you WILL need.
Their forums are a big help and active, users are working to expand the capabilities of these NAS and the programmer is working on making that easier too. Check it out, I've not found anything better yet and with some of the newer versions of SAMBA in the code it's pretty fast too! Perfect for a HTPC but not so great for a big transactional database
http://www.lime-technology.com/
Build it, Drive it, Improve it! Hybridz.org
We make S3D movies. Lots of them, in HD, IMAX, 2K DCP, etc. Tons and tons of data, with lots of revisions, none of which any creator ever actually is willing to get rid of.
Our server array is (mostly) made up of machines each of which has a 14-bay case, a fairly basic mobo with 6 onboard SATA ports, two added 4-port PCI-E SATA controllers for a total of 14 x 2TB = 28TB per server for less than $1K not including drives. Our servers happen (for stupid red-tape reasons) to run W2K8 Server but any SAMBA equivalent would work just as well.
No redundancy or backup though: this is only for (easily reproducible) render output. Key stuff is kept on systems with RAID-5 arrays, nightly mirroring, etc.
No. Well...maybe. Actually, yes. It really just depends.
Preferring Western Digital drives (for no particular reason) I have a pair of 1TB My Book Essential Edition external USB drives as well as a 2TB My Book World Edition network drive (which I got form a guy for like half the price).
Anyway, the World Edition has a USB port that allows me to connect the other two drives to it using a USB hub and it shows them as network shares in addition to its own folders.
Another nice thing about the World Edition is that it runs Linux so there's neat stuff you can do with it, mine is currently running a torrent client called Transmission which has a Windows (only a wannabe Linux geek) front end I can use to control it remotely.
http://en.wikipedia.org/wiki/ATA_over_Ethernet
This is something I've always wanted to play with. It's a little expensive (for a home user) to get into, but it's extremely scalable. If I moved all my DVDs and such to on-line storage, I think this is what I would opt for. It can be run in all sorts of RAID configurations, doesn't require matched sized hard drives, and it can all be racked up very nicely.
One other thing you should consider, especially with a lot of people recommending dedicated servers, is power consumption: the bigger and heavier the box, the more you're going to pay in monthly power bills. This is one reason why using an old computer that's sitting around and stuffing 6 HDs into it might not be an optimal solution: if it costs you another $10 - $15 a month in power, you can relatively quickly spend your way out of whatever savings you've nominally achieved.
4 drive bay, USB, FW400/FW800 and eSATA. Will take 2tb drives, RAID 0, 1, 5 and 10. Comes pre-populated or unpopulated, the latter is what I got and added my own drives. http://www.macsales.com/ No financial connection, just a satisfied customer (they have great tech support!)
This is obviously not a build-it-yourself storage array, but is a good option if you want a commercial out of the box solution.
Have another script that you run to index things. Basically, a dir /s command [add filesizes to the end if you can]. There's your index of where everything is. Use grep to access it quickly, or load it all up in a text editor and find to access it slowly. I like grep. Regular expressions help.
-Clio
Karma: Bad (mostly from not giving a fuck)
Blog: http://clintjcl.wordpress.com
drobo pro =)
Even if you "need' a 16000 RPM drive, just make it for your local drive that you play your videos directly off of. Use 5400 for all the other ones. Just move your file before watching it. Sure, if you're an impatient baby and want to watch something within 5 seconds of it entering your mind, then you might have to wait 5 minutes if the file is 4.5G. Then again, it's the type of waiting you can go pee or make your snack during.
-Clio
Karma: Bad (mostly from not giving a fuck)
Blog: http://clintjcl.wordpress.com
1) Cheap tower server + your favourite unix distro + software RAID + many, many cheap 2TB drives.
2) Standalone NAS device. Everyone so far seems to recommend different makes so I'll carry on the trend and suggest Thecus. Just slot in the drives and you're ready. Install the SSH module and you also have a Linux server too.
An Atom N510 board with 4GB, a small USB flash drive, and FreeBSD 7.3 or OpenSolaris. You'll get an awesome amount of capacity with negligible setup or maintenance, all while being able to tell whether or not (with certainty) there's data corruption occurring. An old full tower with 3.5" bays going the full length of the front would be the ticket. Be sure to have a powerful enough PSU.
I'm still kicking myself over getting rid of an old Gateway 2000 case: it must've had room for 10 3.5" drives; more, if you'd use drive adapters for the 4 (5?) 5.25" bays.
You can't beat it for price/performance/storage, and it's the same basic thing you'll find in a higher-end SAN type device.
It should be obvious that you need a NAS. Buffalo do devices called Linkstations that can house a single drive storing up to a terabyte (or put in your own disk), or they have Terestations I think they're called with network attached RAID storage. These are all extremely quiet and unobtrusive when compared to doing this with PC hardware. They can be accessed over SMB, NFS, FTP or with other file transfer protocols.
What you do after that depends on how geeky you want to be. I have Freelink running on my Linkstation (Debian for ARM basically) running a full Samba Windows domain with authentication, completely automatic printing support where you don't even have to do any manual printer driver installs and the kitchen sink.
Do you really need to save all those Blu-ray rips of the latest Hollywood blockbusters? Just delete them after you watch them. That way you'll have plenty of room for all those raw uncompressed video.
I'd highly suggest you check out unRaid. It's an inline expandable raid system which allows up to one drive to fail without losing data. I've been using mine for quite a while and I love it!
Track your TV Shows with your iPhone - FREE
1TB or 2TB external USB 2.0 drives are cheap.. just the other day LaClie had a 1TB external drive for approx $70
Get a 6 port USB hub... now you can scale your storage.
If you need a backup.. get another drive.
If you have a PS3 install PS3 Media server (free) and you have a cheap HTC solution.
http://freenas.org/
Booting this off USB flash with 4 1.5 TB drives in a RAID 5 configuration =
It can even spin down the drives when they aren't in use.
Each 1.5 TB drive : $100
AMD Misc processor, motherboard with 4 SATA, GigE, RAM: $200
Case and Decent power supply: $100
I download a lot porn, and I also record a lot of masturbating videos.
Upload you masturbation vids to get free porn!
It would be like the old MP3 trading FTP sites. then site operater puts sells the vids on a amatur pornsite. Where's my Business Method Patent?
If you want cheap, affordable storage get:
A decent full tower case with a modular PSU
A motherboard with 8+ SATA ports (cheap)
A 4-port SATA expansion card
=
12 SATA slots + 12x SATA power for cheap
Get a cheap bunch of 1.5 TB drives for up to 18TB total. If you say home I assume you don't mean 99.9% redundancy. You can buy a new PSU or motherboard or whatever and have it delivered and that's okay. Softraid two drives in RAID1 for 1.5 TB less storage. If you need more protection then upload it to some offsite backup - any external disk or second machine is still vunerable to theft, fire and whatever. It works for me, though I only have ~10 TB due to due of old low-capacity disks.
Live today, because you never know what tomorrow brings
48 SATA drives onboard a 4U rack, dual six-core opterons, redundant power supply. This thing won't let you down. Go big and heavy with this and it'll cost $1/GB. http://www.oracle.com/us/products/servers-storage/servers/x86/031210.htm Make sure to ask for a discount, only suckers pay full price for Oracle gear.
You specified quiet and hidden (small), not cheap, so I'd go for a NAS device the synology DS1010 can do 5x2TB (8TB with redundancy), and if you need more it can be expanded with 5 bays more.
A cheaper option would be to take some old hardware and toss a NAS distro on it, but I'd expect more hassle and noise from that solution
any sort of network accessible drive is going to be relatively slow. if you are copying large files that will be important. if you expect to use the large drive for your working sets, as opposed to just for storage, that will be crucial.
the truth is that you probably won't be happy with anything less than a eSATA interface.
I use these:
http://www.synology.com/us/index.php
Depends on how far you want to scale.
"If any question why we died, Tell them because our fathers lied."
Here's a case I've seen suggested.
http://www.servercase.com/Merchant2/merchant.mvc?Screen=PROD&Product_Code=CK4020&Category_Code=4UBKBLN
Norco has a similar one; I'm not sure about the exact differences.
http://www.newegg.com/Product/Product.aspx?Item=N82E16811219033
You can pick up SAS controllers at the usual sorts of places, newegg, supermicro, whatever. Watch out for the newer couple series of DELL PERC controllers since I hear they've started restricting you to only use them with dell branded drives.
I don't yet use the above types of enclosures, but may pick some up for a work project soon.
As for OS -- just look at FreeBSD 8.0/8.x with ZFS or maybe OpenSolaris with ZFS if they ever release a public B138 / 2010.03 release or better (which are both now long past expected due).
Both should be thoroughly checked under load / production simulation conditions for stability and FS integrity before trusting data to them. Since you deal with large (video) files and not millions of small ones you probably won't be using anything too new / fancy like dedup so you should be safer without the more esoteric / bleeding edge FS features in use.
I'd suggest getting three or more chassis and populating each with about 4-6 1TB HDDs in RAIDZ or RAID-1 mirrored pairs to start with, and set up a redundant cluster of NAS shareed volumes from each of these drive pools from each of the servers such that any one of the server machines itself could totally fail and the other 2+ servers would redundantly have your data preserved via use of a second level RAIDZ/mirror off of the server pool exports. There's discussion about this on the zfs-discuss mailing list archives lately as I recall as well as on the freebsd zfs list IIRC.
Use dedicated 1Gbit LAN links between the servers of course.
If you really don't want to sysadmin an OS look at FreeNAS or NexentaStor's community edition or OpenFiler and see if any of those suit you, but given your space/performance needs, I suspect just running an OS on the servers will work.
As for the PC motherboard, you don't need anything especially high end, a mini-ITX like the D510MO or one of the Zotac ATOM boards from newegg will get you a CPU and motherboard for $80-$200, though usually getting in on one of the fry's weekend bundle sales of a CPU+microatx motherboard will often get you a CPU+Motherboard in the $30-$100 range. All that matters is having enough of the right kind of slots for a SAS controller or whatever you want to use (SATA for lower end), and of course a slot for a gigabit NIC if the motherboard doesn't already have enough of those for you.
Supermicro / tyan have some serverish motherboards if you're interested in those, but usually those are closer to $250-$350 and proabably don't do you much good. Just put 4GB or 8GB RAM into the things and ZFS will be happy running under a 64 bit OS.
Look at putting your cache/ZIL on the main "head" server onto a SSD if you want better performance.
The backblaze design is interesting, though I don't personally see that it is needed at the scales you're talking about which can be readily enough handled with an off the shelf case with a few drives per case and multiple servers for redundancy / backup.
Tahoe-LAFS may be of interest to you if you want to check that out, though it sounds like ZFS itself should just work for you.
Anyway read the freebsd-zfs and zfs-discuss related lists, they're full of good into on DIY SAN/NAS solutions.
2 TB are getting quite cheaper..
I used to have a 4U rackmount with 8 JBOD drives + a master drive, etc, etc, etc. that I built up over two years wasting about 3K on it. It died.
I ended up replacing it all for about $1000 shipped (1.7T Raid 5). It can support 4 2T HD's and
The NAS server is the 409+ non-rackmount from Synology (same-as/replaced-by the 410):
http://www.newegg.com/Product/Product.aspx?Item=N82E16822108050&cm_re=synology-_-22-108-050-_-Product
http://www.synology.com/us/products/ds410/index.php
I spent quite some time researching the HD's to use and settled on these:
http://www.newegg.com/Product/Product.aspx?Item=N82E16822152185
The newegg support was quite nice in helping m get drives from different batches (overkill, but nice).
It has worked fantastic. Supports timemachine and itunes naively, is a media server (integrates with my segate freeagent theater+ and PS3 seamlessly), the download manager rocks, and all our photo's are served up to the extended family (yea dyndns integration). Also integrates with UPS, external drives, and broadcasts the UPS issue to all my machines on the network. support for everything (and I mean everything.)
the admin interface this thing comes with is fantastic (linux on it with busybox and the ability to add your own packages):
http://www.synology.com/us/products/features/index.php
little box seriously rocks.
You've filled your bays, now it's time to fill your motherboard slots with eSATA cards, and start hooking up externals.
My case is big enough that I ran out of internal connectors, and so I slapped an eSATA card in, cut a hole in the computer case, and routed the cable from the outside port to inside the case where I still had three bays open.
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
Get a Drobo S http://www.drobo.com/products/drobo-s.php it is the best
I hate to state the obvious, but I'm going to anyway.
Go buy a NAS. Plug it in. Turn it on. Set it up. Move on.
Unless you find storage fun and sexy, that is. I don't, but different strokes for different folks.
I personally have a little ReadyNAS Duo in my basement, attached to the network. It's almost silent, small, and has a ton of different things you can turn on, and once it's set up, it just works.
It's only (only!) 1TB, mirrored on two drives. According to Netgear, if you want to move up to something bigger you can pull the drives out of it and put it in one of their larger devices later.
My only complaint with it is that it has a 100Mb network interface. I assumed that, being modern equipment, it would be Gb and didn't check the specs, so shame on me I guess.
The preferred solution is to not have a problem.
Has anyone mentioned Drobo yet?
www.drobo.com
Keep your drives as cool as possible. I think there was even a story posted on here a while ago about cool drives are less likely to fail.
I have a RAID 5 made of 4 2 TB drives. I was able to find a case that didn't stack the drives too close together. The case also has mounts for additional fans at the front. The airflow is great and it isn't loud. The house at 72F/22C, and the drives report that they are running at 82F/28C. Also, you will want some way to monitor your drives.
http://www.newegg.com/Product/Product.aspx?Item=N82E16816111051
The cool things is to have windows that bounce up and down like a good tits.
You know I have been considering a business idea of making something that would respond to this query.
It would be a box (similar to a an eggshell tower box, I suppose) with a bunch of drives in a RAID configuration, with an GBit Ethernet or SCSI (if short distance) connection to a PC. It would have an affordable CPU and open source OS.
So basically it is an off the shelf plug-in mass storage solution. You can just plug it into your PC and then mount it as an external drive. And then you have your multiple TB redundant storage, right under your desk.
Something like this is being sold to enterprises for 10s of thousands of dollars, but if I can get one done for the consumer lever that is $500-1000 (based on the number and type of hard drives in it), do you think it will work? Would you buy something like this?
Just add harddrives everywhere, and add them together in a virtual folder. Buy 2TB sata drives internally and USB external SATA docks. Mirror them pairwise if you like, but don't do RAID. Just adds more problems than is solves.
Lets say you have 4 computers that will take 6 SATA 1 ESATA and 2 PATA, and with a dual USB sata docking station, you will have 22TB/computer, so you will have 88TB total for all computers. Now add docking for your USB storage off your routers for maybe 90TB. Copy an entire 2TB HDD to a docked HDD, and store offline as many as you like. A bookshelf holds easily 1000TB.
Each computer runs a script to share the drives on startup, so they are all available easily.
If you like, you can add truecrypt to them all as well, and include this in your script.
The scriptS may look like this with Win: /quit /volume c:\lib\disk1\libM /letter M /cache y /beep /q /v c:\lib\disk2\libN /lN /b /q /v c:\lib\disk3\libO /lO /b /q /v c:\lib\disk5\libP /lP /b .....
MOUNT.CMD
"C:\Program Files\TrueCrypt\TrueCrypt"
"C:\Program Files\TrueCrypt\TrueCrypt"
"C:\Program Files\TrueCrypt\TrueCrypt"
"C:\Program Files\TrueCrypt\TrueCrypt"
SHARE.CMD .....
net share tvM=M:\lib
net share tvN=N:\lib
net share tvO=O:\lib
net share tvP=P:\lib
UNSHARE.CMD /delete ....
net share tvM
DISMOUNT.CMD /dismount /force /beep /quit
"C:\Program Files\TrueCrypt\TrueCrypt"
don't cut it off www.mgmbill.org
I've been running off an Acer EasyStore H340 for about a month. I'm very happy with it; it's very cool, quiet (if anything else is making noise, you won't hear it; in a closet, you definitely won't hear it) and plenty fast for most households. 4 hotswap SATA bays, eSATA, USB, and GigE. I can push 75MB/s via NFS to and from it (reading and writing from RAM), which is plenty for streaming video. It comes with Windows Home Server and it's headless, but I popped the drive into another box and installed Ubuntu with an SSH server. Worked like a charm. I'm also running a 2TB software RAID1, mt-daapd (iTunes) and squeezebox servers. I'll probably put Samba on it too.
The only thing I'd change is that a dual core Atom would be better. I actually haven't run into a bottleneck yet, but I wouldn't try reencoding videos on it. I believe the dual core model will be out this month. No affiliation with Acer; I'm just geeked because this is just the quiet, cheap server I've wanted for years. Sounds like sharing your other computers via NFS (automount) or CIFS plus one of these would address your needs; if not, maybe the info will help somebody else.
I have a Thecus n5200.
5 drive bays, two gigabit ethernet connectors, runs linux on the backend and the community is growing as far as custom hacks. Software raid, and provides iSCSI, samba, NFS and all sorts of other goodies.
--- perl -e 'printf("%s\n", pack "H*", "7369670a676f6c677940676f6c67792e6e65740a2f736967")'
I use a basic Core2 box with 8 1.5 tb drives on a pair of 4 port sata controllers. I'm running Windows Storage server as an iSCSI target and have yet to be disappointed in capacity or performance considering it was pretty cheap to build (using the hardware raid and cheap 4 port cards) I have 9 tb useable storage, and the whole build cost me less than $1200 dollars when I built it and I could duplicate it for less today....
Have you tried streaming porn, from the internet, instead?
http://alternatives.rzero.com/
I have one of http://support.wdc.com/product/download.asp?groupid=901&lang=en>these. It's small, headless, 1TB (the one I have, they have up to 8TB), 1Gbit, and RAID 5 out of the box (but can configured however).
I judt got a nre Kinesis keybiartf so please excusr ant egregiou typos.
I download a lot of 720/1080p videos, and I also seed a lot of pirated video. I have run out of slots to put in hard drives across two computers. I need (read: need to upkeep my ratio) my files at all times (over a network is fine), especially since I mirror TPB, what I've got on the Beowulf cluster. I don't want to have the MPAA beating down my door, I want all hard drives available all the time to anyone connected to the Internet. I'm assuming that, since it's copyrighted material, I won't need more than three lawyers to defend me and thus I'm hoping a solution exists that can be easily disposed of and/or hidden away from the authorities and still keep somewhat untraceable. So Slashdot, what have you done? Fixed that for ya.
UNIX: Find it, fsck it, forget it.
20 SATA slots on a backplane
Definitely the cheapest, look on newegg.
Do you need so many fucking movies? If you do, at least buy them, idiot. Then you won't need so much storage.
Two drive bays, just stick hard drives in them and you can use them as eSATA or USB:
"eSATA USB to SATA External HDD Dock for Dual 2.5 or 3.5in Hard Drives" by Startech ... approx 50.00 on Amazon.
Even with USB, you could easily use 1.5 TB drives and get 3 TB. Need more, just pop a drive out and stick a new one in. Need 6 TB? Get two docking stations.
Going to build something yourself?
"External AC Power Adapter for SATA drives" ... approx 13.00 on Amazon.
And just to throw a USB thing out there, even though you said you didn't want it:
"USB Extender over Cat5 Cable 50-meters 150-FT Extension" ... approx 18.00 on ebay with free shipping.
I just ran USB to my wiring closet with a set of the above (about 35 feet away). Speed seems to be just like it was directly connected. Run to a wiring closet and use a USB hub and you can easily have multiple TBs of storage online all at the same time. No pile of drives setting around on your computer desk.
I've spent a lot of time the last week or so looking at this. I came up with USB over CAT5 to my "wiring closet," the Startech dual drive docking station, and a 4 port USB switch (like an A B switch for USB). With this setup, I can hook up 4 computers to 3 TB of storage. I only use one computer a lot, but if I need to access the drives on another computer, I just go and push a button on the USB switch.
PROs:
The Startech does eSATA
Get a docking station at work and you can easily take a drive to work and not have to mess with cables.
Easily add more drives for the cost of a 2.5 inch or 3.5 inch SATA drive
Cons:
Only 2 drives online at a time unless you buy more docking stations.
I have to go to a closet and push a button to access the drives from another PC.
Have to have extra CAT5 to all comptuers to use the USB over CAT5 adapters
I looked at some USB auto switches, but they are all windows proprietary.
Also, be real careful looking at cheap USB to NAS adapters. On most of them, they can use NTFS as read only.
Doctors destroy health, lawyers destroy justice, universities destroy knowledge, religion destroys spirituality
yawn just buy a proper chassis and a raid card or three. or buy a san/nas those are your choices its pretty simple
I wanted to do a similar task myself this is my solution from 6 months ago:
total cost under $1000, more than half of that was for the harddrives
mobo with lots of SATA ports (6+)
low power AMD CPU (but make sure it scales up, ZFS uses a lot of CPU cycles under use, mine is 1.0Ghz to 2.4Ghz dual core, but only 20-40 Watts)
1 Tb Hitachi Drives to fill it with (surprisingly seem the be the most stable option these days)
1 old hard drive to use Ubuntu and software with (I use a 60 Gb)
Ubuntu and zfs-fuse.
netatalk and avahi for Mac access, also use ssh and svn for important file storage and access, but I'm a command-line junkie.
low power system (less than 100 watts), very low noise (invest in good CPU/case fans)
To save room in my small apartment, I also use this as my main desktop, upgrading my previous 7 year old computer.
Notes on ZFS:
Amazingly easy to use and expand data with. Great backup features. Stress free storage.
Very slow though with database and random access files (photo management/websites/etc),
so I recommend you use your active copy of these on the standard drive.
4 of these
http://www.newegg.com/Product/Product.aspx?Item=N82E16822145276 (or whatever your fav is)
1 of these that supports 4 drives
http://computers.shop.ebay.com/Servers-/11211/i.html?_nkw=1u+server&_catref=1&_fln=1&_trksid=p3286.c0.m282
and then install this
http://freenas.org/freenas
$800 4 TB server (raid 10)
i hate making anonymous coward posts - too lazy to log in
go to addonics (www.addonics.com), they have a lot of awesome gear
then get to know zfs
no more problems. scalable, redundant. its just easy. and OPEN SOURCE you windows fruit cakes :p
Here is a decent recipe for a fast secure home server: Start with an ASUS Maximus III LGA1156 motherboard (with 10 Sata connectors), an Intel I5-660 CPU (32 nm, with support for AES instructions). Add 2 Gb of DDR3 memory, and at least 6 WD Green 2TB drives. Install either a 4GB USB key or a SSD drive for root storage, and then install Lucid Lynx 10.04 Server AMD-64. The latest kernel in Lucid now supports per-CPU threads for both RAID6 and dmcrypt, vastly accelerating both functions. It's now possible with this CPU to easily exceed 100 Mbytes/sec over the Gb Ethernet (using NFS) when running the following software stack: RAID6, with Luks (dmcrypt) using AES-XTS mode, and finally EXT4 (or XFS if you prefer). I've noticed significant bandwidth improvement using Lucid Lynx, even with relatively low performance AMD Athlon dual core cpus. I'm now bonding dual (or triple) ethernet interfaces, because I can saturate a single Gb Ethernet port. This solution still lacks checksums on files, but when BTRFS is released for general use, it ought to be excellent.
Linux based, simple, samba and NFS support.
Nice cheap consumer device.
If I wanted more I'd consider the DNS343, 4 2TB drives =8TB, which is pretty reasonable.
I recommend to you use a NAS, like Openfiler or freenas. Both run in a very simple computer. In a LAN the performace is great. Test it and share it. You can share throug SMB, iSCSI, FTP, RSYNC.
I've used a Norco RPC-4220 Rackmount Server Case which has 20 3.5" Hotswap SAS HDD bays. I picked up a cheap HP refurbished P800 SAS RAID controller and SAS Expander from an auction site. Has worked out really well and ended up being relatively cheap and easy to setup.
I ran into this issue when my wife started taking up photography, and we started moving the DVD collection to my home computer. I resolved it by getting a couple of 750 GB SATA drives, setting them up RAID1, and a couple of 40GB PATA drives set RAID1 for system drives. I use Ubuntu server 8.04, an share the folders with Samba. A 100m network is sufficient to watch movies from the computer attached to the TV, while she edits her work. I back this up to a couple of 1 TB external USB drives. The only time this system has gone down is if I put it down, or we had a power outage that ran the UPS dry.
It's You and I against the World... When do we attack?
I just recently set up a FreeNAS box. using an old p4 that I had laying around. Used a 100gb hard drive for the OS and installed 2tb hard drives in a RAID 1 array. I can use it to stream media or as a backup for my work. Cheap, easy and effective. I am looking at getting an Mini itx motherboard and setting up a 1u rack to save space using this idea.
I personally use a Drobo (www.datarobotics.com) for my video storage. It's highly adaptable to storage needs, and is surprisingly fast, small, and quiet for what it is.
30TB? 60TB? 100TB? For home use? The first data center that I worked in, one of the largest in the world, had about 1TB of 3380 disc drives plus a warehouse full of tapes.
Try Supermicro 44 drive case in 4U, SAS multiplier backplanes, 4G cache RAID controller, and filled with Seagate 1TB drives.
I have a 42U high rack in my room (sane people would put it in the garage) with various hardware in it. Of noteworthyness is the antex 4u22atx case, can fit seven hard drives in it, most motherboards I see only have six or so sata connectors anyway.
Assuming 1.5tb drives, that's 9tb right there. Want more space? add more units and use one of the many clustered filesystems available on linux. Scales up to 100tb easily, well excepting that you'd need more than one power circuit for it all before you even manage to fill the rack.
Network storage is easy. It's not hard to build a NAS solution with 6-20 TB of storage. The cost keeps going down, the software smarter.
The question is, how do you do apparently online access without paying to spin 10-20 hard disks 24x7? The green side of it interests me more than the tons of storage side. At the end of the day the energy cost to keep them running is worse than the cost of the storage.
What OS will spin down the disks but somehow keep enough info cached to make access only a question of disk spin up for final data access?
The solution I'm thinking of keeps a segment of every file buffered and only those disks with the buffers are kept spinning; the disks with the whole file are spun down until needed. The buffer is big enough to allow access to the data until the disk with the data spins up, mitigating the access delay. The whole storage system is automatic, determining the appropriate buffer size based upon disk spinup delay and creating buffered segments and maintaining the smallest possible buffer disk count possible.
Bonus points for a RAID-like parity system that keeps the smallest possible parity sets to minimize the number of disk spinups required.
The final solution is of course solid state storage, but it'll be another 5-10 years before SSDs are cheap enough and dense enough to consider.
Ya it's windows... BUT it's easy easy and you can toss all your old drives into the box with no issues. I tried the NAS stuff out there and some linux solutions. End of the day linux is to squirrelly and the NAS stuff at the consumer level is to slow at the disk.
WHS gives you the flexibility to build your own server to fit your needs. Me I needed super fast transfer over the network to move the files so I built to suit. One of the best additional awesome things that this gives you beyond a place to store and serve all your video is a kick ass way to back up all your computers that is dead simple. Ya, it's not raid, and ya it's not linux but I have yet to have a problem and it is like two clicks to restore files or entire back ups. Video playback with a cheap video card is awesome if you go hdmi from the box right to the AV gear with no headaches with codex and what not if your video is multiflavor.
WHS FTW.
LG just put out a nice line of 2 and 4 Bay NAS. I picked up a 1TB, 2 bay NAS(1TB x1 HDD) with DVD-Rewriter and iSCSI support for $200. Has Itunes server, can pass it torrent files, print server, 3 USB ports and a media reader (no CF though..). LCD panel with menu system and buttons for limited computer-less operations like backups from flash drives, etc. 20MB/s writes, up to 40MB/s reads, I love it. My old Linksys NAS 200(4MB/s read and write) is now doing duty as the offsite backup at my in-laws house.
Good-bye
Get a Drobo. It's a RAID cabinet that holds up to 4 SATA drives (newer models hold 5 or 8 drives). It's hot-swappable. You can add or replace drives without shutting it down, and if you need more space, swap a 1TB drive for 2TB.
You don't specify what constitutes lots of data. In my case, 2 years ago I went for 6 750GB SATA drives in a Raid6 configuration. There's some very good posts here about some lesser known data reliability options, but personally I wanted to go with a worldwide standard that had been around for a long time and wasn't reliant on a couple guys hacking code in their spare time to make disk redundancy and file access work.
I bought a standard full size tower case, got a very large power supply, and spent a good deal of money on a mid-tier Raid controller. My primary requirement was Raid 6 so I could lose two drives without losing all my data, and my secondary requirement was having true hardware raid support. Most Raid controllers that are not enterprise business class are not true hardware raid - meaning that they use software and the CPU for some of the operations. This slows down file read/write. I did the research and read reviews and got a decent Promise card - if you have the money, go for LSI, Areca, or 3ware. Next, I got a Promise hot swappable 4 drive SATA bay. Not really sure why, it doesn't serve any purpose since in 2 years I haven't had a failure and thus have not had to hot-swap a drive. A very important thing is that I also purchased 7 drives for my 6 drive setup. So I already have a spare if I need it, and I don't have to worry about having the spare cash when a drive fails, or waiting on an RMA if it was still under support, etc. The one thing I wish I had done, and still might, is buy a spare raid controller with the exact same chipset. If your raid controller fries, ALL of your data is gone unless you can get the array up and running on an identical controller. That's a freaky thought!
6 drives in raid 6 at 750GB gives me a little under 3TB of disk. I wanted that in a single partition for ease of use, so I messed around with some 64 bit Linux distributions and did not have any luck. I finally settled on Vista of all things, but only after I got fed up with fighting with Linux - I didn't give it a fair shot, I should have been able to make it work. The only thing I can think is that it didn't like my controller or motherboard.
So, 6 drives of 750GB in Raid6 gives me 3TB. At the time I had less than 1TB of stuff, and wanted to make sure I had room to grow. I didn't grow anywhere near as quick as I expected, and I'm still at less than 2TB today. 2TB drives in a raid6 would give you 8TB, and that's if you only used 6 drives - you could easily add more into that same Raid6 array (depending on how good your Raid controller is). Even if all of your movies are dual layer quality, say 6GB each, that's over 1300 movies. That'd certainly last me a long time!
Daisy-chained firewire drives.
If you are on a PC, most cards have multiple ports. Even on a mac with a single port you can put a lot of drives in line.
Note I started on SCSI busses and 10-base2, so daisy-chaining came naturally. With termination voodoo no longer needed it's a lot easier than it used to be easy.
Big storage really isn't hard - it's the backups that come out to bite you.
I recently did a few solutions for a photography customer!
The Drobo Pro + 8 2TB Drives (green 5900RPM drives so not performance oriented), on newegg at the time of the sale I got it on: $2200, the was STUPIDLY simple to setup, has network AND usb/other connection types, has some pretty sick beyondRAID functionality (RAID5 spinoff by drobo)
Currently setup 8TB full already... he cant stop thanking me... (he had 6 different 1TB+ external drives)... I really with i had the money to get one for myself... Q_Q
So Slashdot, what have you done?
8 2TB hard drives in a RAID-6 configuration running Mandriva Linux 10.0, and using a Pentium D and motherboard I had laying on the Pile Of Unloved Parts. Its not the quietest solution, nor is it the greenest, but neither of those points was my concern. I was more interested in the ability to have two drives die and not lose files.
I want a new quote. One that won't spill. One that don't cost too much. Or come in a pill.
External SATA. You can also find little connectors to turn internal SATA ports into external SATA ports.
No, I will not work for your startup
They are cheap, fast, and reliable. IMHO Synology makes the best ones. I've had a DS409+ for over a year and it provides 6TB wherever and whenever I want it.
Backblaze uses, and specifies (it's not a kit, BTW - just a parts list) , which is among the design choices deemed 'flawed' by many.
If it suits your needs, great. Protocase and Newegg would like your business. If it doesn't make the 'quality' cut, SuperMicro sells nice stuff.
If not, a suitable JBOD solution hunt isn't exactly news.
...specifies
consumer grade discs - , which is often what people criticize.
Redundant power is the second - addressed by commercial JBOD hardware, the price difference of which can pay for your discs or upgraded disc..
Preview, itomato, preview.
I've posted this idea before, but I would like to see a harddrive jukebox, so you don't have 20 drives running 24/7, just a couple.
Basically your could plug 20+ hard drives into what amounts to plastic holders for the harddrives.
You then have a jukebox/cataloguing software like DVD jukeboxes or some sort of virtual drive that keeps track of all the files on all the drives. When you need to access a file, it powers on and spins up the correct drive. Ideally two or three drives could be activated at once. The Jukebox software would automatically handle all of that.
As far as how the connections were made: You could do it a couple of ways. Have each drive with its own SATA connectors that all fed into an electronic switching hub that handled activating the drives. Or you could have it be a phsyical/motorized scenario, where each drive plugged into a custom SATA header that then interfaced with a motorized SATA connector to attach to a specific drive.
This has already been done with tape drives & DVDs obviously, but I am talking about something CHEEEP (the extra E is for extra cheap).
http://blog.slaingod.com
Sounds like you need a Drobo (from http://www.drobo.com./ The various units are expensive, and you'll probably need the Drobo FS if I read your post correctly. The upshot is, though, it's expandable to 20 TB of space. Just shove a drive in.
(Note: Not a Drobo vendor, just a fan who wants one himself)
--
# Canmephians for a better Linux Kernel
$Stalag99{"URL"}="http://stalag99.net";
Or an Xserve RAID that can hold 14TB - rack w/o drives can be had for about $500. 14 slots. Dual power, dual fiber channel, dual battery backup, dual ethernet.
There are two types of people in the world: Those who crave closure
but to answer your question: build a drobo
I bought one of these when it first came out, even before Newegg got on the bandwagon. I got it because it's capable of RAID5 that is transparent to the host and doesn't require port multiplication. Similar things in the same price range that preceded it all required a host that was port-multiplier-aware, which the eSATA port on my motherboard is not. This box uses a new JMicron JMB393 chip that can create RAID 5 within the confines of the box, making the drives in it appear to any host as one big physical disk. There's no software RAID driver; any OS (that is GPT-aware) will see the drives in it as a single disk. There is software, but it exists to manage the internal RAID system, not implement it.
I popped five Western Digital Caviar drives into it, told it to create a RAID 5 volume across all of them, made sure my host was ready to handle GPT (Windows XP-32 doesn't), and then voila, a 4TB virtual disk that has some redundancy. The JMicron management software has a juvenile UI (q,v. Asus software) and needs some work with regard to its SMART awareness, and the two fans are bloody loud for a desktop, but it does what I need it to do and I could afford it. I got the five WD10EADS drives for about $70 each, and the Venus T5 cost me another $220, so 4 terabytes of standalone redundancy for less than $600. You might build a cheap NAS with software RAID 5 for less, but that isn't what I wanted.
Check out Western Digital's EVDS drives. The 2TB ones cost about $10 more apiece then their other 2TB "green" drives. And they're "slow". They're also fucking huge and perfect for multimedia streaming; it's like you're paying extra for something that's underclocked. Supposedly, they make up for it by very low power and more reliable.
But remember, no matter what kind of drives you get, you'll get them in pairs and software RAID1 them. Period. (Well, unless you're doing something fancy like ZFS. Ignore the people who tell you that you have to ZFS, though. But do be redundant, somehow.) There isn't a single manufacturer who doesn't have complaints from some user, "my drive died 3 months after I got it." That doesn't mean they all suck the same, just that shit happens and it might happen to you.
The one problem (as with any solution here) is that 10TB is nearly impossible to back up.
That depends on whether all that data is changed or is only archived. For archiving 2 4TB external drives can be used to store most of the data with a 2TB drive used for data that is frequently changed. What is more problematic, because it takes action, is to transfer old data to new storage and test it.
Currently I've got a Mac and I use Carbon Copy Cloner to backup my internal HDD to external drives but I want to use Ubuntu as well and don't know how it will work with files backed up by CCC. CCC is supposed to do an exact copy but when I tested a backup once I noticed it didn't preserve all of the file metadata such as creation/modification dates and file attributes or permissions. There are 3 user accounts on my computer and each user was able to open files other users owned. Maybe I didn't use the right options when running CCC. So, will I be able to preserve the metadata and if so can Ubuntu work with it as well is something I'll have to find out. However I am able to make 1, 2, 3, or more backups some of which I can keep onsite while archives are kept off-site.
Falcon
Should there be a Law?
I use a RaidZ ZFS pool with 6 1TB USB drives for about 4.5 TB of total storage. The drives are plugged into a dedicated UPS. The computer on it's own UPS will power down before the drives so I have assurance all data will be committed to disk if I lost power. It was easier then getting a PC with a power supply strong enough to power the drives. My PC has 6 USB ports and gives about 50MB/s downloads. When I used a USB hub I achieved 15-20MB/s which is still good enough for video.
DroboPro ($1200 after price shopping and rebates) and eight 2 TB WD drives ($119 ea. specials). Has built-in RAID, 2-drive failure tolerant, and Gigabit Ethernet. 2-drive fail setting nets about 11 TB usable, 1-drive fail nets about 13 TB usable.
Try Western Digital My Book World Edition... its great!!! and you can access it by internet and wireless.... really god!!
I've got a $20 case, $50 500W power supply and $40 motherboard with SVGA and ethernet, its 5 PCI slots each stuffed with $25 4x SATA cards, 20 $100 1TB HDs. Running Linux, network mountable drives and ssh login.
That's $210 PC + $2000 HDs for 20TB. That's a lot of porn storage for you.
--
make install -not war
http://www.sansdigital.com/
the TowerRAID series offers various configurations to meet your needs and is easy to setup.
I have two 5 bay boxes and an 8 bay box and I am very happy with them. They come with controller cards so all you need is a free pci-e slot.
The bottleneck in your scenario is the network, so I/O speed isn't an issue. I have an old, full tower sitting in the garage doing your job with a multi-disk software RAID.
I actually think an LVM might work out better.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
The main use for a usb stick on freenas is to store the OS .. which last one I did like that was 128MB flash drive. Old pentium-2 computer from 1998, floppy disk to boot into the USB, then all the drive bays open. Add some PCI cards for additional spaces.
My last hardware build a month back was taking an old P1-100Mhz tower case (one of those with the huge number of bays), pulled the motherboard and put in a P2-400Mhz motherboard like this.
Somewhere on the net was a guy that used stacks of usb hubs fully populated with usb flash drives. Must have had 80 usb drives attached. Raid system set up. Proof of concept. Cool like the guy with a dozen node Beowulf cluster in a plastic toolbox.
Why not just get a motherboard with 8+ sata slots, a cheap processor, a full tower and load some *nix on it. If you really need that many HD's I think that is your best bet without getting some professional disk array setup.
You gave a bunch of links which require wading through hundreds of lame arguments, including 100 different people proudly proclaiming "RAID is not a backup solution". Nobody wants to read that shit.
http://www.lian-li.com/v2/en/product/product06.php?pr_index=331&cl_index=2&sc_index=5&ss_index=13&g=f
Lian Li ex-50.
Supports up to 5 disks.
Comes with a 2 port E-SATA II PCI-e 1x card (SiI3132 based), nice power supply and big cooler. Supports RAID 0 / 1 / 3 / 5 / 10 / JBOD / Port Multiplier.
Not bad for $200.
Afs is overkill, it's nice to share files over the internet, it's nice if you want to secure files from your sysadmins., it's nice if you want to upgrade to paranoid security levels.
However once something goes odd it is not your friend. When there is time skew figuring out that it's freaking out over time isn't trivial. When your distribution changes from mit to hemidal kerberos or reverses-- things can go bad. If you you aren't familiar with many of the security checkpoints in your OS things can get less than pleasant.
firewall off your stuff, pop up a whole slew of RO and RW NFS mounts, make automatic softlinks to the files. For the digital editing, copy to the mounts as needed.
oh yea.. same directory structure on all shares. make a copy script that copies the file to the right directory on the drive that can fit it best.
ie: transfer terminator2_judgement_day.mpg /action/scifi
and it will move the file to a mostly full drive if possible. into the /action/scifi directory on that drive. and then softlink the file into the /action/scifi directory on your browsing directory.
This allows for some nice backup options as well.
CoolerMaster CM830 Case
4 3into2 SATA hot swap bays
3Ware 9500S-12ML 12 channel controller
12x 1.5TB SATA drives
2x 6 drive RAID5 - 6690GB each
GF8200A / Phenom X4 965 / 4x2GB RAM
Intel 80GB SSD X25M boot drive
external USB DVDRW (no point tying up a drive bay)
Windows 2008 x64 (2008 R2 not supported on 9500S - it's an old card, but works for me)
Some other VM's for things like my Exchange server, etc.
Not terribly complex, but has been reliable now for ages, and it works great.
Cheap solution: - Buy an atom-based motherboard with SATA and PCI slots (Since it's a home server, you probably won't need a beefier CPU.) - Put SATA cards in said PCI slots - Put it in a huge case, invest in hard disks You can easily get a NAS with six SATA drives that will run at less than 200 watts (you can save a lot there if you have drives spin-down while idle, but I'm told this reduces their lifetime.) I have a similar machine set up in my closet serving up 2TB of space to my network. It costs about $5/month in energy to keep it on 24/7.
Check this Wayback Machine http://www.archive.org/iathreads/post-view.php?id=243665
I'd like to buy homeland for our 10 million people. http://twitter.com/mahadiga
This puppy can be had for 200 bucks and hosts 8 disks off 2 SATA ports. You can easily host two of these with practically any PC that has 4 internal SATA ports as brackets are readily available. I have even seen low profile brackets which carry 4 SATA ports to the outside via an SFF-8470 connector http://www.pc-pitstop.com/sata_cables_adapters/4X7PLLP.asp and then a breakout cable http://www.convertermarts.com/servlet/the-166/SAS-SFF-8470-4x/Detail is again readily avaible. Even Mini-ITX boards tend to have 4 SATA ports these days. Drobo clone recipe based on ZFS http://pegolon.wordpress.com/2009/01/13/build-your-own-drobo-replacement-based-on-zfs/
The only sensible recommendation one can make is getting a NAS like this one: http://www.qnap.com/pro_detail_feature.asp?p_id=109
It's by far the fastest, most hassle-free and ecofriendly way of supplying any reasonable amount of storage. It does RAID5 and RAID6, iSCSI, ext4, 2x 1000MBit/s Ethernet, Ajax web-based management interface etc., so all the bases are covered well.
This client does video and film conversion to digital storage. They archive the customer stuff for months in case the customer needs a redo or something. So they eat a fair amount of storage.
For archival purposes, they couldn't spend tens of thousands on a SAN or anything like that, so I pitched a Burley (also known as MacGurus) enclosure a la this: MGBurly8PM - Burly 8 Bay w/Port Multiplier http://www.burlystorage.com/ccp0-prodshow/MGBurly8PM.html with 8 750GB hard drives (the biggest around a couple years ago). This is an expensive enclosure but well designed. With today's hard drives, you can cram 8TB of storage in it for another $800. It's worked very well except for one hard drive which had to be replaced with a 1TB drive, and another component which we had to replace when it failed.
Piece of advice: do not try to mix and match port multipliers, enclosures and hard drives. This is a recipe for failure. There are issues with the multipliers firmware, the drive firmware and other bugs. Buy the lot from one source and make sure they test it before they ship it to you. Burley knows what they are doing and we've gotten great support from them.
We also wanted to serve up some 8TB of space to some Macs for video editing using iSCSI. First we bought some Micronet external enclosures, two enclosures with four drives each. The Mac drivers didn't work at all with the Macs, and Micronet had ZERO support for that issue. We tried using the boxes on the Windows XP video editing machines - Adobe Premiere 1.5 (ancient, I know, but the client can't afford to upgrade yet because they're using older Matrox video cards) does NOT like that kind of thing at all. So I took one of the older XP machines and put OpenFiler on it to serve up the space as iSCSI for the Macs. This has worked very well. Final Cut Pro uses the iSCSI storage without any problems at all. OpenFiler was installed, configured and has not been updated for the last year or so and it has faithfully served up iSCSI storage to four Macs without a single hiccup (except when a staff member plugged in an air conditioner to an overloaded circuit and dropped every machine in the room - restarting the OpenFiler box, it went right back to work.)
To archive and serve up a bunch of videos, just get a big enclosure (or several smaller ones), load them up with big drives, hook them up to any reasonably cheap box with a couple GB of RAM and just serve up that storage any way you want. OpenFiler can do it.
One caveat: Do not use Seagate drives. Their quality control these days absolutely sucks. We just installed a HD video machine recently with four 1TB Seagate drives in a RAID 0 array, and one of the drives started giving read errors within the first few days. Reviews of the Seagate drives over 1TB on most of the retail sites like Newegg show either DOA or clicking within six months. They're selling Seagate 1.5TB drives for $125, which is a good price - except the customer reviews are dismal. I wouldn't touch a 2TB drive yet with a ten-foot pole, by any manufacturer, until I see good reviews on the retail sites - and I mean a hundred or more reviews with 75% or more being 4 or 5 stars.
Contrary to one person's viewpoint above, Hitachi tends to get good reviews. I'm going to be using a new one this weekend to back up my stuff on my home machine.
My two cents.
Richard Steven Hack - This sig is TOO GODDAMN SHORT TO DO ANYTHING USEFUL WITH! MORONS!
Sorry. I'm an Apple sysadmin, but I still cringed at what you just said. 1, he's a home user that wants a quiet and cheap solution. 2, Apple doesn't make these anymore, so when they die, you're stuffed. 3, Promise way outperforms these bad boys in every single way now. What's the point?
Creating a RAID 5 array on OS X, is really not that hard. On Windows, pretty trivial too, never got around playing with it on Linux though, must be pretty straightforward.
Personnally, I prefer SOFTWARE RAID, because it's easier to recover from a failed array compared to a hardware card that is not manufactured anymore.
(example, take the drives to another MAC or Windows machine, they will be recognised automagically if it's done with the OS, not so if the controller went belly up...)
I'm running a raid 1 on an old G4, and RAID-5 on an old Win2k Server machine, and they are easier to manage than the one in my Proliant (Hardware)
I've got better things to do tonight than die.
http://www.willudesign.com/BlackDwarfTop.html
I have been where you've been, and let me tell you, what you are considering is a waste of money.
Moore's law applies particularly well to harddrives. Every two years you can buy a new hard drive that is twice as big, for the same price. Or pay half as much for the same storage capacity. If you stock up now, you'll spend a lot of money for something you can buy a lot cheaper in two years' time.
My advice: Buy new hard drives and replace them as you run short on space. If you run out of space inside your rack, move the contents from your oldest disc into the next, and you can sell, discard or get an external enclosure for your oldest disc.
Allocating massive storage without immediate need for it, is going to cost you a lot of money.
Seriously, how much ????
FragHARD or don't frag at all
Internal SATA/PCI card. They'll have at least two, and probably more sockets for SATA2 drives. Easy, cheap and efficient. I've got an old IBM Intellistation...cost me $250. The boot drive is SCSI. There's an open SCSI connection. It has 2 IDE sockets (which would accept 4 drives) and one SATA socket. I bought a PCI-SATA2 card with two more sockets. That's all I need. I could have got one with 4 just as easily, and almost as cheaply. With a slightly newer configuration, I could have done PCI Express - SATA2 even cheaper.
So if I wanted to just install hard disks, I could put 9 in. For the bucks, you can't beat it. And twin processors keep things plugging along nicely. It isn't terribly fast, but you can't make it grind.
I've calculated my velocity with such exquisite precision that I have no idea where I am.
I think that was his entire point. The questions have all been answered and analyzed, and answered again. The answer isn't hard to find if you know how to look.
Now, there are several answers so there were several links. The differences depend on the specific needs and budget allowances. What the op asked isn't anything new. The only thing new was his use.
You gave a bunch of links which require wading through hundreds of lame arguments, including 100 different people proudly proclaiming "RAID is not a backup solution". Nobody wants to read that shit.
They have a name for it: 'slashdot'
One of my colleagues owns a lot of movies in BluRay format.
Since it's a PITA to access his collection, he simply rips the Blurays and reencode them in high-quality.
I don't have the details about the quality, but I guess an encoded movie is around 2 gigabytes with 5.1 sound.
This way, all his collection is available on a single disk on his network.
Unless you are a top BT uploader of BDRip movies, I think you don't need to keep your collection available online.
Use tapes or DVD to store them.
I am the happy owner of a homebuilt fileserver since a few months. I just went shopping for an energy-efficient CPU and motherboard (MSI's active phase switching really seems to help) with enough SATA ports and exentension slots (this is where Atom failed, even including price). I got a number of EcoGreen disks (they do 5400rpm) and a separate noteboook disk for the OS. I got a case to fit it in (i have a AeroCool M40; quite a nifty case but having 5 disks + a OS disks seems to be a bit many already), and assembled it.
I installed Linux with normal ext4 for the OS and made a raid 5 over the data disks with software raid (mdadm). The filesystem i then put on the raid is XFS. It performs quite well and allows easy capacity upgrades might i ever need it. It cost me a bit over 500 euro for 5.5 TB of usable storage (exported via NFS, Samba, FTP).
ReadyNAS
Around fall 2009, I went to NewEgg, searched motherboards by SATA ports and found a gigabyte mobo with 10 SATA ports that uses AMD chips. Intel's a bit too pricey. Around the same time, I got a reliable PC Power and Cooling PSU for $100. During Black Friday 2009, I picked up 10 2TB hitachi drives for $110 each. The gigabyte motherboard has 2 ethernet ports so it works great as a gateway/file server. 20 unformatted TB (14.3 tebibytes after raid6 for 2 drive failure redundancy) for about $1500.
No dealing with expensive NAS cases, expensive ass sata/raid controllers (lol 4 port sata controllers that cost $120 for a decent brand vs gigabyte motherboard with 10 sata ports for $150).
I write out the binary data on all the surfaces of my house (furniture included) in 0.000016 point font (using a portable electron microscope) and with a special array of mirrors can access the data from anywhere in the house. I however did once make a typo when recompiling the data for a home video and my son ended up with Angelina Jolie's nipples for eyeballs. other than the compile/ decompile time it a relatively painless system.
This was a solution created for a customer. It works well, it scales well and it is as cheap as the hardware ou need to buy.
You need to get a PC with a mobo that supports VMDirectPath and has several PCi-e available. Then, install VMWARE ESXi 4 to a first small hard disk (and old one, for instance).
You can then buy an external enclosure or just create something yourself and you just keep on adding HDs and SAS or SATA controllers (wichever you like).
(Take care to avoid HD vibration, tough - It's bad for a HD's performance.)
Make your first VM Open Solaris and give direct access to the controllers (and consequently the disks) to that VM, and make sure you give it CPU and RAM priority.
You can then setup a ZFS filesystem using RAID-Z or RAID-Z2 (much better than traditional RAID 5, etc). Solaris and ZFS make it easy to make it available as NFS or SMB or iSCSI.
You should worry about speed and the bus speed limitations as you add controllers and disks, but on the other hand if you need more availability than speed this should not be a problem.
Basic hardware to build this should be no different than your average home computer, and you can use any mb, if you'd like, instead of one that supports VMdirectPath, by using raw device mapping, but you should take into account that it is not supported by VMWARE.
Also, because Open Solaris is given direct access to the HDs and ZFS is thus created directly on the hds, if you plug those disks into another PC or VM running OpenSolaris, you access all data on your array with a single command line - Hardware Agnostic. (Usefull if your HD dies).
This has the advantage of letting you raise whatever other VMs you may want. Of course, you can do this, installing open solaris directly. ZFS is the way to go.
Note: You can use Nextenta Store free version if you plan to use store than 12TB of Data. Easier to set-up, but same principles.
Send your address to RIAA. They will solve your problems for a nominal fee.
Why on earth would you need to watch video faster than the actual speed of the video? I checked a random 1080p video file I had on my hard drive (yes, porn) and it has a bitrate of 2 megabytes/s. Randomly googling for HD tests I found that the difference between a slow drive and a Velociraptor is that you can watch a 2h movie in 8 minutes with a slow drive and 4 minutes with a fast one. Yet you suggest waiting the 8 mins _before_ starting to watch the movie? Didn't it cross your mind that if you can copy it from a slow drive to a faster one in 5 mins, you can watch it directly from the slow one assuming it's actually more than 5 mins of video?
I have a server that acts as the backend for MythTV as well as the household fileserver. I use a 4U Norco case that I've been quite happy with. My case is quite old and only has 7 3.5" HDD bays (6 of which are kept quite cool) and 3 5.25" bays. The newer version of the case has room for 10 3.5" bays directly in front of two fans. My current rig started with 5 400GB drives (mix of SATA and IDE). One by one those disks are being replaced with 1TB HDD in a software RAID array. I'm using samba and NFS for access. You can probably build something like the following easily enough:
NORCO RPC-450TH 4U Rackmount Server Chassis
LGA 1366 Motherboard
Power Supply
Intel Core i7 Quad-Core Processor
12 GB RAM DDR3 1333
10 TB disk space: 10 1 TB 7200 RPM SATA 3.0 hard disks
CD/DVD Drive
My 5 1/4" half-height MFM drives are about the size of a DVD drive and quite massive.
I'm sure some collector out there has one bigger but I'm not jealous. Well, maybe a little.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
I archive all files and collect lossless live reocrdings. On some days I dl a gig or more and this has been going on for 14 years.
I have built home storage tservers, I have used off the shelf products.
Of eveything I have used/created, the best is the following:
http://accessories.us.dell.com/sna/products/Hard_Drives_External/productdetail.aspx?c=us&l=en&cs=19&sku=A3082986 I've ordered some from Dell, the price is better than most and they are quick at fulfilling.
The 4TB iomega storcenter...Comes setup with RAID 5, uses all high quality components/drives....Have a TON of features (like Itunes server, media servers, and many other features. Will email you if it detects an issue or reaches any set point which you define...
I have tried other similar devices, like the Western Digital version of this - it is okay, but is more expensive, has less features, is a lot less eloquent of an interface as well...The Iomega has exceeeded my expectations in every way...
Most of all - it just works....perfectly....And to have a 4TB NAS with all of these features, $599 is a good price.
Here's my setup that's currently capable of holding 44TB of storage (I have 18TB so far). Nothing fancy, Just something to hold all my media that isn't horribly noisy or hot and that was still relatively cheap. I have it sitting on a coffee table in my home office so you could put it pretty much anywhere.
A $320 Norco 4020 case that has 20 hot swappable drive bays plus 2 more fixed drive bays inside. http://www.newegg.com/Product/Product.aspx?Item=N82E16811219021
A $250 server motherboard with at least 2 PCI-X slots. I chose http://www.newegg.com/Product/Product.aspx?Item=N82E16813182142 because it takes a Core 2 CPU and DDR2 and I already had plenty of those laying around so I saved a few bucks in parts. I also had a CORSAIR CMPSU-850HX laying around and used it for a power supply which runs about $190.
$99 SUPERMICRO AOC-SAT2-MV8 SATA card. 2 of them will fill the server if you pull the DVD drive (I use an external anyway). 6 sata on the Mobo + 16 more from the 2 cards. Some people complain they're slow but I can pull 60+ Mb/s over the network from them. My guess is they're putting them in regular PCI slots on regular MoBos and not PCI-X slots on a server board.
For an OS, I simply use Windows Home Server. It's $99, windows simple, and is perfect for just storing video files. Reinstalling the OS can be a massive pain though as WHS reinstall script thing never works when there's a controller that WHS doesn't support out of the box (ie. the Supermicro cards). And the new version of WHS based on WS 2008R2 is on the way and there won't be an easy way to migrate.
I also use Flex Raid (Software Raid 4) and sacrifice one disk as a parity drive because duplication isn't much safer but eats a lot more of my space. I just have it do the rSync when no one is likely to be doing anything with the server so it's never a hassle.
So the base cost is within a few hundred bucks on either side of a grand. Less, if you have parts that can be cannibalized from old machines
From there, I just add Western Digital Caviar Green 2TB drives as needed. They're cheap, quiet and run cool. The EARS ones need a jumper (and none are included with the drive) to run under WHS but are $15 cheaper than the EADS ones on Newegg. And of course you can use any old drives you have laying around too.
I have Acer Aspire Revos ($330 http://www.newegg.com/Product/Product.aspx?Item=N82E16883103235) running XBMC Live installed on a USB Drive (xbmc.org) hooked up to each of the TVs in the house and use wired Gigabit (provided by whatever the cheapest 5 port Gigabit switch was on Newegg at the time, I think it was about $25) to stream DVD ISOs from the server to the TV. I don't have much HD stuff and the Revo can get bogged down when you try to play really high def video. 10Mbps works fine with XBMC using vdpau, but an 18Mbps MKV was a bit too much and went slideshow in places.
2TB-4TB isn't "massive" in my book.
I've had an external eSATA 5 disk array for 4 yrs now. I'm looking at an 8 disk external array now - with 2 eSATA ports. The enclosure is $250 including power and fans. I prefer using an existing Linux server with software RAID. There are many reasons to prefer software RAID over RAID cards these days.
So you end up with two RAID sets - 4x1.5TB in RAID5 becomes 4.5TB usable. You get 2 of those, so 9TB available "online." For the rest of the storage, I'd buy an external eSATA/USB drive holder - not an enclosure to quickly swap 1-2TB disks with specific media in and out of the server. Be certain to get both an eSATA PCIx card and eSATA holder that support hot swapping with your OS of choice.
Next, how do you backup all this stuff since we all know RAID is not a backup. DVDs are simply too small for this amount of data. BluRay may be the most cost effective, but you'll still want par2 files included to help recover from bit rot of optical media. Or just always purchase disks in parts and keep 2 copies of everything with 1 off line on a shelf.
Last, You probably want to convert the MPEG2 media to more efficient formats like x.264 or xvid. That will save 50%-75% the storage just with better video compression.
For example, I just recorded Entrapment in 1280i OTA. That file is 8.4GB in size. The xvid version at 720p resolution WITH 5.1 AC3 sound is 1.05GB. It is a beautiful encode, if I do say so myself. The movie isn't THAT good. ;)
My media player does play HD MKV, but I haven't learned how to make them as part of my automated conversion process. With the Lucid Lynx update, x.264 encoding was broken on my install, otherwise I'd be using it. That has been part of my conversion process for many months.
With all this media, you'll want a good system to access and locate it. Windows Media Center is not the answer.
Good luck.
For 130.00 US get one of the new plug computers with the ESATA interface.
http://www.globalscaletechnologies.com/p-32-guruplug-server-plus.aspx
I purchased a 2TB external drive with USB and ESATA port from buy.com for 130.00 last week.
Use Samba and NFS to export your volumes on your local network to your various computers.
I have an older plug computer connected to the external drive via USB 2.0 and it streams movies and music to a variety of devices in my home.
Add multiple NAS boxes.
I have a similar setup to previous post. Server running Linux with a 3ware RAID card running RAID 5. 8 500GB disks with ~3TB of usable disk space. This has been running flawlessly for over 5 years. I have a a movie collection, music and network share for 3 HTPC in the house. Works very well but I wish my RAID card supported the ability to power down the disks and save of power/heat when not in use.
I am almost at capacity on the RAID volume, so to expand I have another RAID card that I can put in the server and create a new volume or replace the 500GB drives with 1TB or now 2TB disks. Replacing the disks would save power and heat, but I would need to backup and restore 3TB of data. Adding another RAID card is easy, but crates more dives that I can't turn off and eat up power.
I am actually thinking about building a new server with the thought of being able to add an many SATA ports as possible (via SATA cards) and then us port multipliers. The use a software based file system or RAID that allow me to add drives of different sizes to the volume. Similar to ZFS but more open. This would make growing the system much easier and allows me to power down drives when not in use. I would still get plenty of performance for my needs.
The other thing that I am doing that most people don't think about, is I backup my entire NAS to another server. I took another old PC that I had and put a 4 port SATA card in it and four 1 TB disks and run Linux and software RAID on it. Each night it powers up and runs a script to back up the primary NAS. I do this just in case something catastrophic happen to my primary NAS and I also use it when I moved to larger disks on the NAS previously. I use rsnapshot to look for changes on the primary NAS' file system and only back up data that has changed. It also keeps the lat 3 months of files that have been changed or deleted, if I need to recover something. When the script is finished, it powers down the backup NAS and wait until the next night to run again.
Iv got a win server 2008 r2 box running as a Dc,dns,dhcp, wsus with an areca arc1230 12 port card. 2x2tb in raid1 for mega important data and 10x2tb in raid 6. It all works great nice io of around 100 r&w the card has 1gb ram too. :-)
My main desktop at home has been running Ubuntu Linux for the last 6+ years, and for the last 3 years I migrated the functions of my NAS (which used to run on a separate linux box) to it, so I have one box less to administer, to consume power, or to break down.
I have currently 10TB of total disk space using 5 2TB disks (8TB usable, when you account for the RAID5 redundancy overhead), but could easily migrate to 20TB on 10 2TB disks (16TB usable, and gaining extra redundancy by moving from RAID5 to RAID6). The result is:
- Very upgradeable (starting with 250GB disks back in 2004, I've migrated all the way to 2TB disks, and will continue doing so; the old disks are simply replaced with the newer/bigger ones and re-purposed as off-line storage, being plugged on a eSATA dock when I need them
- very fast (as the disks are on the machine itself, it's much faster than acessing the files on a NAS over the network);
- very usable (it's all mapped on a couple of ReiserFS filesystems, created on top of LVM volumes, directly accessible without needing to mount anything or configure anything over the network);
- very reliable (I'm protected against any one of the five disks failing, thanks to RAID5 configured on top of Linux MD, through for more disks RAID6 is really recommended, and ReiserFS in my experience is very reliable against crashes and power outages, at least *much* more reliable than EXT3 and XFS).
- very cheap: I've used the SATA controller already available on my motherboard, providing for 6 SATA disks, and apart from the disks I only had to spend money on a multi-disk internal rack: these are great, they fit 5 SATA disks on 3 x 5.25" bays on the front of your desktop, are very cheap (around $75) and give you hot-swapping and great ventilation (via a large, low-noise fan in the back) to boot. Just be sure to use a computer case that has no "rails" or other protuberances between each 5.25" bay, or else you won't be able to insert the rack as it spans 3 bays.
You could use a SATA RAID controller (or even SAS disks and a SAS controller), but I found that it's quite expensive, and unnecessary as the above setup gave me all the speed I needed, and them some.
In short, I'm very satisfied with my setup, and I recommend it to anyone who has large disk space requirement at home.
Some pointers to the hardware I'm using:
- Motherboard: Asus M4A78-EM which is reasonably cheap, very stable and has 6 SATA ports (5 internal and 1 external), fitting the bill perfectly;
- Disks: Seagate SATA 2TB 5900RPM Retail kits : The retail kit (instead of the bare OEM drive) gets you a disk with FIVE years of warranty (instead of just 3 years) and comes much better packaged (so reducing the chances of early death due to shocks during transportation).
- 5-disks-on-3-bays internal SATA enclosure: NORCO SS-500 : great little bay, as described above. - External eSata dock: Startech SATADOCKU2E: with it, when I replace my old (smaller) disks with new big ones, I can re-purpose the old ones immediately as off-line storage,very efficiently (my motherboard already has an eSata connector) and very cheaply (I store the disks on plastic storage cases when they are not docked, very cheap and compact.
Hope the above is of help.
Best Regards,
Durval Menezes.
I have never met a computer that didn't like me.
We have a similar problem at the Mad Lab - we generate a lot of data (from our digital studio and other projects), we need access all the time, and we need reliable storage.
At first we were putting lots of drives into a PC -- but that led to problems. For one thing there was a single point of failure (main board, power supply, take your pick). Another problem was that the system was loud and power hungry. Then there was the backup problem -- there was no efficient way to do it without building another system just like it --- you can't ship TB of data off-site via the 'Net for backups, it just isn't practical. Then to make matters worse we decided we couldn't do anything else with the server without putting our data at risk... that was the last straw for me -- The server was overkill for the task and couldn't be used for anything else. I was stuck in a paradigm - I knew better - but I'd forgotten that temporarily...
Then I hit upon the solution of using D-Link 2-Bay network storage devices. http://www.dlink.com/products/?pid=509 These little guys are reliable, solid, efficient, and affordable. They're also pretty green because they will spin down when you're not using them thus saving power and POH time on the drives.
We use them in pairs: NS0 for storage, and BS0 for backup. With a firmware upgrade they will do NFS - so we have one of our servers map the two devices and then rsync NS0 to BS0 once per day. Newer versions may have the NFS capability built in (it was on it's way, it was beta firmware when we did it).
Now we have the key features of the high-end NAS solutions we use in data centers, but we have it on the cheap. The solution is scalable (more storage, more enclosures), reliable (mirrored drives all around, fast and easy to access (NFS or Samba - take your pick), provides redundancy (outboard power supply for each enclosure - easy to swap, separate controller for each pair of drives), and easy to manage (what's not to love about a scheduled rsync task via nfs for automated backups?).
We can easily access the data from either windows or *nix boxen on the internal network without any trouble. When we need to access the data from outside the Mad Lab we shell into a server and sftp what we need from there.
Here is a pic of the two of them in the rack: http://www.lifeatwarp9.com/wp-content/uploads/2010/01/MadRackBefore-225x300.jpg We've been using this setup for quite a while now and it's been dead solid. When we need to expand we just plug in another unit and map it. We've also installed this kind of configuration in customer facilities to manage backups and solve other storage problems on the cheap.
_M
Anyone who works with computers has heard questions like "Jim changed my spreadsheet last week, can you recover my original file?"
Or, in the case of TFA who is a home downloader, how would he feel if his copy of Humphrey Bogart's "The Enforcer" had been overwritten by another movie?
You could just buy the video, instead of ripping people off, you worthless twat.
I have a clunker PC running FreeNas in my closet loaded with 1 TB drives. Works like a champ. The PC was free, FreeNas is free, and the drives are $100 each at my local parts store.
I don't respond to AC's.
I have an aging Dell Poweredge tower with an Adaptec AAC-RAID SATA PCI card that's equipped with hardware raid. I run 3 500gb drives in RAID-5, so that comes out to 1TB. I have one port to spare if I ever want to expand. I have a bunch of ripped movies, backed up software, and my desktop documents/music backed up there. Ubuntu is the operating system. Don't make the same mistake I did by trying to go cheap and run software RAID. It failed every time the drives were put under a demanding load (i.e. viewing movies or running a backup). When you run 3 drives, the PCI bus runs out of bandwidth very quickly. I wouldn't spend a lot of money on a NAS from HP, Dell, Iomega, etc. They're typically overpowered for backing up. You don't need an assload of RAM or a quad core to do what you're trying to accomplish. I have a Pentium III 1ghz processor and 256mb of RAM and have zero problems.
Chewbacon
The Bible is like Wikipedia: written by a bunch of people and verifiable by questionable sources.
If you have the cash to spend ($600 without drives), you'll make an investment you won't forget. Drobo FS would be my recommendation. You have 5 slots of drives (so you can store up to 10 TB with 5x2 drives), it has BeyondRAID so your files are always safe across all drives, and it's network accessible. Plus, even if two drives fail at the same time, all your data is there. Pretty nuts, but we're going to get one at work and I'm saving up for one at home. http://www.drobo.com/products/drobo-fs.php
I just built a Linux software raid server for my media. Its a mere ~7TB right now but it was ridiculously easy to setup and I dont know anything about Linux. I used Ubunut and the mdadm utility. Started off with 3 1.5TB drives and im at 6 now with 2 more waiting to be added. Making your array expandable is pretty important considering your media collection will only grow and the mdadm method seems to work very well. If you were to get an actual expandable NAS (drobo, netgear ready nas etc.) you're cooked if the NAS itself fails. So far I have moved the hard drives between 3 machines and almost no re-configuration was necessary. I installed ubuntu to a 4GB flash drive, and have 2 SATA cards for the drives to make it easy to move around. The first installation was on a Pentium D 2.6GHz system but I realized I it only had a 10/100 NIC so transfers were slow. My second machine was an Athlon X2 7750 machine (with GNIC) and all I had to do was move the flash drive with ubuntu on it, and move the 2 PCI SATA cards and BOOM it booted up with the array mounted. Since it was a new network card I just had to share the array again and everything was back to normal. The third, and current, machine is a small form factor P4 small form factor PC (paid $80 for it) thats sitting in a closet at the back of my house. To access the media I use a W7 desktop, WinXP laptop, a Macbook, a WDTV Live and a Mac Mini with XBMC on it. Everything works flawlessly and I havnt had a problem. I've written a step-by-step guide that I gave to 2 non-linux users and they were both able to setup without a hitch. Let me know if you want it and ill send it to you.
USB can support up to 255 devices on a single bus. With a USB hub and a few 1.5+ TB USB hard drives, you'd be set. Your absolute storage limit would probably be somewhere in the 500 TB range, after which you could just by another box, and repeat.
The society for a thought-free internet welcomes you.
NetGear ReadyNAS. Relatively simple, very reliable, five year warranty, relatively inexpensive. I've had one running for several years. Works great.
Unless you are acting in a professional capacity, you don't need so much disk space.
Most of your photogaphs are rubbish, as are most of your home videos, so prune and keep only what is worth keeping.
As for the DVDs, CDs and MP3 tracks that you are backing up, most likley you will not revisit 99% of them (how many times can one watch a favourite movie or hear a song?), so remove anything older than one year.
And thinking about doing this in a professional capacity, you should have backup to secure media (i.e. tape) anyway, current projects will fit in normal disks, medium terms projects can be kept online just in case, older projects should be backed up properly and then removed from the disk area.
I run 4 D-Link DNS-321's, two with 2x 1 Tb drives, and 2 with 2x 2Tb drives, all in a RAID 1.
I have 6tb (soon 8 as I finish the next round of upgrades) with redundancy. With a GbE I can run video to all the PC's in the house, both Windows and Linux.
Each case represents about $300 of hardware, so it's not dirt cheap, but you can build up to it, and add on to it fairly easily.
I'm not crazy,I'm actively irresponsible.
I won't need 16,000 RPM drives [...]
You're in luck... They don't exist. =)
Thank you, Edward Snowden.
"Arguments from authority are worthless." —Carl Sagan
i think better than this you cant do
http://www.dasprids.de/blog/2010/04/24/raising-private-storage-to-the-limits
its a friend of mine and he is pretty data crazy
So if you got the money go with that :)
http://www.chenbro.com/
you know you can fry stuff putting things into things that dont like the things you put into it...
http://www.drobo.com/ Problem solved
Here's my suggested strategy for the MPAA:
1. Buy out all the hard drive manufacturers.
2. Pressure ISPs into providing truly unlimited downloads.
3. Secretly seed loads of torrents with 1080p copies of all the movies they can find.
4. Wait for the geeks to run out of disk storage and buy TB after TB of disk space.
5. Profit!
Not knowing for sure what you are capturing this "uncompressed" video content with, and why you believe it needs to stay uncompressed makes any recommendation here 25% effective at most, probably closer to 7%. I understand that it is your money but in my opinion unless you are Jim Cameron and these are scenes to Avatar II there just isn't a great reason to store uncompressed, certainly not ONLINE. Most of the mezzanine formats are non-standard and not very open or supported on multiple platforms, however, you should look into AVC Intra. AVC-I 100 looks awesome. I have been involved in testing at several major content suppliers and tests have proved it to be every bit as good as DNxHD 220 and ProRes 4:2:2, it is definitely worth looking into. Adobe, Apple, Autodesk, Avid, Grass Valley, Panasonic (obviously) and others either already support it, or will be there soon.
For those out there trying to make the math work, AVC Intra @ 100Mb/s would be ~15:1 compared to uncompressed 720p or 1080i (1.485Gb/s). That is where the 7% above comes from.
This thread has lots of good suggestions for storage. I have a distant business relationship with Drobo, and think they're an interesting choice. I have a Windows Home Server as well, and find it to be a step-up from my previous Buffalo Terastation NAS box from a reliability and performance standpoint. I happen to also have a full-size tower and appreciate the simplicity of throwing lots of hard drives at the problem.
However, as formats switched from DVD to Blu-Ray and equivalent HD content, the economics shifted IMO as well. Given the both my Sony TV's, as well as both Tivo's can stream on-demand HD directly (Sony does it without annoying buffering by the way), and considering that I only rarely watch a movie more than once or twice, it's actually become more economically feasible to simply rent HD on demand for $4.99 a shot. There are still going to be a few Blu-Ray discs worth buying to own the content, and more than half of those seem to ship with a free digital copy for import into Window Media Player or iTunes. Even if you own all your HD content on disc now, it's probably worth your while to look into a hybrid model where you rent what you have a passing interest in, and buy/store those few things that either aren't available on demand, or that you have a more long-term interest in retaining.
Oh, and for the porn, a 2TB drive in a large tower should be more than sufficient. Windows 7 and Bit-locker full drive encryption doesn't impede HD playback on a reasonably speedy system. Although it is debatable whether or not 1080p is actually a good thing in some cases. Anyone want to go into business with me creating a unique line of porn-star body make-up to deal with pimples, waxing irritation, and razor-burn?
Too easy, problem solved. Next one?
(This brings feasible maximum to around 34Tb RAID-5, with still 1Gbit/s connectivity per hard drive. If ~330Mbps is enough you can go for 104Tb, RAID6)
Pulsed Media Seedboxes
Why not use a network attached storage device? (NAS) - check out Qnap.com and their lines of NAS devices that use SATA drives / these are stand alone devices with RAID that can be hot expanded - We have 3 TS-509's and they are rock solid
So you're admitting to the whole world you're doing something illegal? You're not making it any harder for the RIAA :)
Anyway, in addition to full towers, what you could also do is expand with 8 port SATA controller cards.. as many as you can fit in there. They're cheap, I got mine for about 60 pounds each from ebay last year.
For holding the extra hard drives, get a small HD internal caddie that transforms your front 5'' area, so you can fit say 5 or 6 instead of 4 in the same area. Something like this
http://www.scan.co.uk/Products/Jou-Jye-ST-3051SS-525-(3-Bay)-Backplane-for-5x-35-SAS-SATA-HDD
Any more HDs, maybe external SATA caddies can also work, and you just get a long SATA cable
Finally, contrary to the above, What I also did (for my really important files, or ones so large/rare), is get a Netgear ReadyNAS NV+ (there are 4/6/8 bays). It's worth every penny.. Similar to the Drobo, but I like it because It's one of very few (if not the only) that has full compatibility with Mac (OSX).. including AFP, NFS, SMB, Time Machine backups, etc..
It's weird how I didn't see a post about Drobo. I have the first generation that can go up to 16TB. It may seem a bit pricey but it works with all OSes, is quiet, tells you when you have a problem, and you can just pop in and out drives to upgrade or for dead drives.
I suspect I'm about to be SlashDot'ed... My own solution was to build a custom case using two full-tower cases, and several (now 7) 3:5 bay hot swap cages. With two 16 port RAID cards, I have a total of 35 bays, 30 of which are covered by hardware RAID. The project wasn't cheap, but it could probably be done for about $2,000 today (using previously-owned RAID cards), and it gives a temendous amount of growth potential. I initially had two computers in the case, but I eventually reduced it to just one and added more bays. The only problem in a home environment is that it generates a tremendous amount of heat. Pictures and information are here: http://scottl.net/projects/orthros
unRAID is by far the best solution out there:
http://www.lime-technology.com/
I don't want to have swappable USB drives, I want all hard drives available all the time on my network.
I use daisy-chained FireWire 800 externals as I only have a laptop. Mine is good for 16 devices per port, of which I have one 800 and one 400. So I'm good for 64TB, although that'd be 32 little PSUs, and FireWire externals tend to be quite a bit more expensive per unit than USB externals and of course raw internal drives.
http://www.readynas.com/
I have a Pro and a Duo. Based on debian, community hack support, all around good deal. I couldn't build a tiny pc with as many features and space for the price. X-RAID means you can keep expanding as drives get bigger and cheaper.
http://packetnexus.com
If you want the "fun" of building your own disk server, then by all means go ahead.
If you don't want the bother, then have a look at Tom's Hardware sub-site smallnetworkbuilder - it reviews NAS boxes of various sizes and capabilities. It provides performance data as well as reviews.
I would recommend getting at least two NAS boxes, and filling them with 2Tb drives. Plan on putting all the data on both NASs, for redundancy. Put all the NAS boxes on UPS.
I use Thecus boxes, but there are lots of options.
Yeah, actually I had forgotten that Promise had picked up the trail, and led it forward. But for the price, an old Xraid is a decent home solution now, and parts are available on ebay & elsewhere. Not sure he wanted quiet - give that he wants lots of drives, which require lots of fans.
There are two types of people in the world: Those who crave closure
Check out the Disks and Storage Systems forum. There are tons of people here with large capacity file server projects. I have my own 15TB box.
http://hardforum.com/forumdisplay.php?f=29
I've been using unRAID for 3 months now, and it fills the bill wonderfully. You don't need matching drive sizes, don't need a RAID controller, don't need powerful hardware, and the list goes on and on. http://lime-technology.com/
For that price you could get a better system like the Thecus N-5500.
Not as hackable but still very modular.
--- I am known for the ones who want to find me on the net. Is that a privacy risk or a privilege? One might wonder..
My experiences using drivebays have been very broad; although found out the hard way, hard drives like to be mounted into a metal cage, to dissipate heat and to protect against shocks and vibrations. Those disks perform a lot longer when having their case cooled; that's one of the reasons Lacie uses metal enclosures to spread the heat.
There are 3.5" drives which have a 2.5" mounted in a cooling block for that same reason.
--- I am known for the ones who want to find me on the net. Is that a privacy risk or a privilege? One might wonder..
It has 20x hot-swappable drive bays. It is noisy, but I keep it in the storage room. Here is a review: http://www.justechn.com/2009/07/02/review-norco-rpc-4220-4u-20x-hot-swappable-drive-bays
I used a 5-in-3 backplate with a cheap RAID-3 card for years. It was going well but I lost the array just as I was planning to upgrade. I considered rolling my own with FreeNAS or similar but in the end went for the better option - a QNAP TS-809 in the loft connected by gigabit ethernet with RAID-6. That will sort you for 12 TiB easy and it's all Linux and hackable.