Slashdot Mirror
Building a 10 TB Array For Around $1,000
As storage hardware costs continue to plummet, the folks over at Tom's Hardware have decided to throw together their version of the "Über RAID Array." While the array still doesn't stack up against SSDs for access time, a large array is capable of higher throughput via striping. Unfortunately, the amount of work required to assemble a setup like this seems to make it too much trouble for anything but a fun experiment. "Most people probably don't want to install more than a few hard drives into their PC, as it requires a massive case with sufficient ventilation as well as a solid power supply. We don't consider this project to be something enthusiasts should necessarily reproduce. Instead, we set out to analyze what level of storage performance you'd get if you were to spend the same money as on an enthusiast processor, such as a $1,000 Core i7-975 Extreme. For the same cost, you could assemble 12 1 TB Samsung Spinpoint F1 hard drives. Of course, you still need a suitable multi-port controller, which is why we selected Areca's ARC-1680iX-20."
One: The title is a borderline lie. Yes, you can buy 12x 1TB drives for about a grand. But if I'm going to build an array and bench mark it and constantly compare it to buying a Core i7-975 Extreme, the drives alone don't do me any good! (And I love how you continually reiterate with statements like "The Idea: Massive Hard Drive Storage Within a $1,000 Budget")
Two: Said controller does not exist. They listed the controller as ARC-1680ix-20. Areca makes no such controller. They make an 8, 12, 16, 24 but no 20 unless they've got some advanced product unlisted anywhere.
Three: Said controller is going to easily run you another grand. And I'm certain most controllers that accomplish what you're asking are pretty damned expensive and they will have a bigger impact than the drives on your results.
Four: You don't compare this hardware setup with any other setup. Build the "Uber RAID Array" you claim. Uber compared to what, precisely? How does a cheap Adaptac compare? Are you sure there's not a better controller for less money?
All you showed was that we increase our throughput and reduce our access times with RAID 0 & 5 compared to a single drive. So? Isn't that what's supposed to happen? Oh, and you split it across seven pages like Tom's Hardware loves to do. And I can't click print to read the article uninterrupted anymore without logging in. And those Kontera ads that pop up whenever I accidentally cross them with my mouse to click your next page links, god I love those with all my heart.
So feel free to correct me but we are left with a marketing advertisement for an Areca product that doesn't even exist and a notice that storage just keeps getting cheaper. Did I miss anything?
My work here is dung.
How is this news? Yes, we all know traditional HDs are cheap. Yes, we know that you can buy more storage then you could possibly need. So how is this newsworthy? It really is no faster nor more reliable than SSDs. I think this is more or less a non-story.
Taxation is legalized theft, no more, no less.
That'll buy the disks. But nothing else. "Hey, look at my 10TB array. It's sitting there on the table in those cardboard boxes."
Sorry, I saw Areca and I threw up in my mouth a little. Their controllers are terrible, and gave our company nothing but trouble in the short amount of time we used them in the past. Those that are still out in the field (sold to customers and have service contracts) are a constant nuisance.
This headline is very misleading. Sure you can buy 12x1TB drives for just under a grand, but you won't have anything to connect them to, as the controller itself is another $1100. Another eye-catching headline to get click through's, that' just wrong. Sad.
What good are 12 hard drives without anything else? Absolutely nothing. An enclosure alone to correctly power and cool these drives costs at least $800 and that's only with (e)SATA connections. No SAS, no FibreChannel, no Failovers, no cache or backup batteries, no controllers, no hardware that can connect your clients over eg. NFS or SMB to it.
Currently I can do professional storage in ~$1000/TB if you get 10TB, including backups, cooling and power that would probably run you $1600/TB over the lifetime of the hard drives (5 years).
Custom electronics and digital signage for your business: www.evcircuits.com
We needed a solution for backups. Performance is therefore not important, just reliability, storage space, and price.
I reviewed a number of solutions with acronyms like JBOD, with prices that weren't cheap... I ended up going to the local PC shop and getting a fairly generic MOBO with 6 SATA plugs, and a SATA daughter card (for another 4 ports) running CentOS 5. The price dropped from thousands of dollars to hundreds, and took me a full workday to get set up.
It's currently got 8 drives in it, cost a little over the thousand quoted in TFA, and is very conveniently obtained. It has a script that backs up everything nightly, and we have some external USB HDDs that we use for archival monthly backups.
The drives are all redundant, backups are done automatically, and it works quite well for our needs. It's near zero administration after initial setup.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
Really? You run 10 TB array on Windows vista. nuff said - http://forums.freebsd.org/showthread.php?t=3689
And get FF plugin to avoid 8 pages clicks http://www.teesoft.info/content/view/68/1/lang,en/
Disclaimer: I'm one of regular contributor and part of mod team @ the official freebsd forum.
Another thing with RAID arrays that have quiete a few drives is, you have no method of correcting a flipped bit. You need at least RAID6 to correct these errors. With such vast amounts of data, a flipped bit isn't that unlikely.
I did someting some years ago with 200GB (and later 500GB) drives:
10 drives in a chieftec Big tower. 6 drives go into the two internal drive cases, 4 go into a 4-for-3 mounting with a 120mm fan. Controller: 2 SATA on board and 2 x Promise 4 port SATA conroller 300 TX4 (a lot cheaper than Arcea and kernel native support). Put Linux software RAID 6 on the drives, spare 1 GB or so per drive for RAID1 (n-way) system. Done.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
+5 Thorough
Ok, so let's say you built one of these monsters. Or you rolled your own with linux and a bunch of drives.... How would a home user, back this up? They've got every picture/movie/mp3/resume/recipe etc.. that they've ever owned on it.
Anybody got any reasonable ideas?
From the .COM bust, I have two leftover Netapp filers, with a dozen or so shelves, about 2T of storage. Each unit was about $250,000 new. A half million dollars worth of gear. Sitting in my shed. It's not worth the cost of shipping to even give the unit away any more. I guess it'll probably just go to the recycling depot. It seems a bit sad for such a cool piece of hardware.
On the cheerier side, it is nice to enjoy the benefits of the new densities; I have two 1T external drives, I bought for $100 each, mirrored for redundancy, that sit in the corner of my desk, silently, drawing next to no power. (Of course the NetApp would have better throughput in a major server environment, but for most practical purposes, a small RAID of modern 1T drives is just fine.)
Love many, trust a few, do harm to none.
Such a RAID is for an always-on server. Expect about 8 watts per drive after power supply inefficiencies. So 12 drives, around 100 watts. So 870 kwh in a year.
On California Tier 3 pricing at 31 cents/kwh, 12 drives costs $270 of electricity per year, or around $800 in the 3 year lifetime of the drives.
In other words, about the same price as the drives themselves. Do the 2TB drives draw more power than the 1TB? I have not looked. If they are similar, then 6x2TB plus 3 years of 50 watts is actually the same price as 12x1TB plus 3 years of 100 watts, but I don't think they are exactly the same power.
My real point is, that when doing the cost of a RAID like this, you do need to consider the electricity. Add 30% to the cost of the electricity for cooling if this is to have AC, at least in many areas. And the cost of the electricity for the RAID controller etc. These factors would also be considered in comparison to a SSD, though of course 10TB of SSD is still too expensive.
Has it been over a year since you last donated to the Electronic Frontier Foundation
I've been using FreeNAS 0.7 RC1 for a while. It works pretty well for a NAS, and does the job for my small business. However, I don't think it would be useful for a larger business that requires great performance and reliability.
A NYC lawyer blogs. http://www.chuangblog.com/
Also notice that they decided to stick desktop drives in a Raid array, a big no-no if you want your array to last more than a few weeks.
Not my experience. I did this with Maxtor 120GB and 200GB drives some years ago and in >3 years 24/7 (then the systems were replaced) I had one failure in 50 drives. (Ok, there were some additional ones, but that were drives dropped in shipping.) I think the "big no-no" is just the drive vendors wanting to earn more per drive by rebranding the same hardware as "RAID edition" with a few firmware changes. At least with Linux software RAID you do not need any "RAID edition" drives.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I mean, who is the target audience for the article??
People who just want massive amount of data storage for private use just buy a few NAS units, plug them in a gigabit Ethernet or USB hub and keep the more needed data on the internal HDD's.
On the other side, people who want fast, reliable and a lot of data storage buy something like a HP Proliant, IBM or similar Rack server with redundant PSU's, RAID controller with battery packs and SAS HDD's at 10-15k rpm (and possibly a tape drive).
The later setup costs more in the short run, but you spare your self a lot of head aches (repair service, configuration, downtime, data loss) in the long run, as this hardware is designed for this kind of tasks.
So who is the article targeted at: wannabe computer leet folks? And why on earth is this article on the Slashdot frontpage??
12 consumer level SATA drives by Samsung. What'd be interesting is to see how long it takes before it fails with complete data loss due to drive failure. Raid 5 isn't going to save this turkey.
I'm pretty sure those are supposed to be TB and not GB.
more reliable if you're gonna make an array?
sure, it's got some sort of leniency but RAID5 means only one drive can fail...and sorry, but I don't think samsung drives are known for their reliability, let alone Areca's controllers.
Plus think of the downtime as it rebuilds the array when a drive does go out. That's definitely gonna throw a wrench in those average throughput numbers.
I'd go for a 3Ware controller and enterprise class drives as they are meant to last longer.
Pictures of the setup up would have been cool, but they didn't do that. This article is dry and useless to say the least.
This is not Linux software RAID though. From TFA, we are talking about hardware raid. The drives don't fail catastrophically, they just have other features which cause the controller card to think they are failed. You can tell the controller to just rebuild the array. This is fine as long as only one drive does this at a time...
Yes the premium paid for RAID edition drives is outlandish, but the reason to use them in hardware RAID configurations is the tighter response times and to other firmware features that come with the drives.
Good point about the controller costs. I have been facing a similar problem with my own massive storage setup. One thing I have found that works well is getting Dell Perc 5/i cards from ebay. New they are around $500 or $600. You can get them on ebay for maybe $125. This allows you to connect 8 SATA drives via one PCI express slot. I've only tried it with FreeBSD and JBOD style configuration though.
I've got half the uber setup they talked about and its works great for me. With 6 sata ports on my mobo and another 2 in a pci-x by 1 slot (I found a regular pci card for $10 with two ports). I've got plenty of space with only an additional $30 on the card. I use mdadm in a raid 5 with 6 x 1TB drives with one spare. One 300GB drive for the OS and I had the rest of the parts laying around. You could assemble the setup I've got for $500 if you have any old system with a large enough case. Add a backplane for another $90 if you you case is only a midsize. I have never benchmarked the speeds but It seems fast. The price was certainly right.
For those who are concerned about backing up large amounts of data. Please call your local data storage company. Yes they do exist, but I'll skip naming names as I don't like to shill for free.
Simply ask them about external storage devices you can use. They'll often lease you the equipment for a small fee in return for a yearly contract.
For 3 years I simply had a $30 a month fee for a weekly backup to DLT tape (No limit on space, and I used a lot back then.). They gave me a nice SCSI card and the tape drive with 10 tapes in a container that I could then drop off locally on my way to work. Did encrypted backups and had 2 months (8 week) rotations with a monthly full backup. With the lower cost LTO drives that came out a few years the costs should be minimal. Can't wait till all this FiOS stuff is deployed. I'm hoping to start a data storage facility.
If you have your own backup software and media don't forget to check with your local bank for TEMPERATURE CONTROLLED SAFTEY DEPOSIT BOXES. Yes banks do have some location with temperature sensitive storage. Some of those vaults can take up to 2k degrees for short periods of time without cooking in the interior content.
Where I currently am the NetOps is kind enough to provide me some shelf space in the server room for my external 1TB backup drive that I store my monthlys on. I have 3 externals giving me 3 full monthly backups (sans the OS files since I have orignal CDs\DVDs in the bank)
For home brewed off site I suggest a parent or sibling in a basement but elevated. I used a sister's unfinished basement up in the floor joist inside an empty coleman lunchbox (annual backups).
Now a days with my friends having sick disk space also we tend to just RSYNC our system backups to one another in a ring A -> B -> C -> D -> A with full backups each node syncing to the next on separate days during the day when we are not home.
PSEUDO CODE
===========
CHECK IF I AM "IT" IF SO
SSH TO TARGET NODE
CAT CURRENT TIME INTO STARTING.TXT
RSYNC BACKUPS FOLDER TO TARGET
CAT CURRENT TIME INTO FINISHED.TXT
TELL TARGET "TAG YOUR IT"
BACKUPS\ ...
A_BACKUPS\
B_BACKUPS\
Put each node's backup folders under a quota if needed to ensure no hoarding of space.
To really crunch the space you could try and pull off doing a delta save of A's backup such that B's backup is the delta of A diffed to the subsequent nodes (Might be important for full disk backups such that a lot of the data is common between the systems).
-=[ Who Is John Galt? ]=-
For backups I really like FreeBSD ZFS (with raidz). I rsync in new data from my servers and then create a ZFS snapshot. It works quite well. I have running it at home and at work. Of course the raidz is software so you can use whatever cheap controllers you have around. The thing that makes me love it is that ZFS does its own checksums so it can detect if the data it is reading does not match what it wrote.
I've done this every 2-3 years three times now for personal use and a couple times for work. My first was 7x120 and used 2 4 port ATA controllers and software RAID5. My second was 7x400 and used a Highpoint rocket RAID card. My third one is 8x750gb and also uses a Highpoint card.
Lessons learned:
1. Non RAID type drives cause unpredictable and annoying performance issues as the RAID ages and fills with data.
1a. The drives can potentially drop out of the raid group (necessitating an automated rebuild) if they don't respond for too long.
1b. A single drive with some bad sectors can drag down performance to a crawl.
2. Software RAID is probably faster than hardware RAID for the money. A fast CPU is much cheaper than a very high performance RAID card low end cards like the Highpoint are likely slower for the money.
3. Software RAID setup is usually more complicated.
4. Compatibility issues with Highpoint cards and motherboards are no fun
5. For work purposes use RAID approved drives and 3Ware cards or software.
6. Old PCI will max out your performance. 33Mhz * 32bit = 132MB/sec minus over head, minus passing through it a couple times == 30MB/sec performance
7. If you go with software RAID you'll need a fat power supply, if you choose a raid card most of them support staggered start up and you won't really need much. Spin up power is 1-2amps typically but once they're running they don't take a lot of power.
8. Really cheap cases that hold 8 drives are hard to find. Careful to get enough mounting brackets, fans, power Y-adapters online so you don't spend too much on them at your local Fry's.
For my 4th personal RAID I will probably choose RAID6 and go back to software RAID. Likely at least 9x1.5TB if I were to do it today. 1.5TB drives can be had for $100 on discount. So RAID5 $800 for ~10TB formatted or $900 for RAID6. +case/cpu/etc...
I'd love to hear others feedback on similar personal use ULTRA CHEAP RAID setups.
All prices in Canadian $$$ Buy 2 of these to fit a total of 8 SATA drives: http://www.canadacomputers.com/index.php?do=ShowProduct&cmd=pd&pid=021484&cid=516.690 (289.98 + taxes) Buy 8 1.5TB Drives: http://www.canadacomputers.com/index.php?do=ShowProduct&cmd=pd&pid=019453&cid=HD.443.877 (1119.92 + taxes) Total: 1409.90 + taxes for 2 external SATA enclosures & 12TB of disk space. Setup takes less than 1 hour. You can always just start with 3GB + 1 enclosure for a total of aprox 450.00 to begin with and keep adding on to it as drive prices go down and as you need disk space...you will save even more that way
Find somewhere you can host a duplicate hardware setup--maybe a friend's place, in exchange for hosting a copy of theirs at your home. Sync them regularly via rsync-over-ssh with --bwlimit so that nobody gets cranky about their web browsing working poorly. This'll protect you against hardware failure, though you might want to do something involving revision control, as noted, to guard against other problems.
Laws do not persuade just because they threaten. --Seneca
Seriously this is not news. it's an advertisement to sell gear that doesn't exsist.
You can get two 6-disk SAS/SATA expanders under $170 each. They require 4 SATA connection in total so a cheap SATA controller or an onboard will be adequate. Then use the software raid of a modern linux kernel, or stuff it with RAM (>8GB) and put opensolaris and a ZFS raidz fs. Bonus extra are hot swappable trays and hardware monitoring.
The problem is that as storage densities increase, the statistical likelihood of a RAID-5 being unrecoverable after a drive failure is getting uncomfortably high. If the controller fails the bad drive, there can't be a single bad block among the parity data distributed among the rest of the RAID set. The higher MTBF on the enterprise drives reduces the likelihood of that sort of data loss; for a lot of people it's a worthwhile tradeoff. Personally I don't trust the figures; I'd go with RAID-6.
Once you start dealing with RAID-6, though, you end up fighting against cost. A 4TB RAID-6 built from 1TB drives will cost you ~$500 for the drives, but a decent controller will set you back at least $400 new. For that $900, you could get 5TB of RAID-1 or RAID-10 using software RAID, and will probably end up with better performance.
Provided you have the controller to cope with it.
But why is this even on /.? Who cares about that personal story?
Or can I do an "article" tomorrow, about the 127 $5 mice I connected to my pc,and how I got it to display 127 cursors and do coreographies with it on a beamer?
Actually I think this would be more interesting than TFA. ^^
Any sufficiently advanced intelligence is indistinguishable from stupidity.
Not sure if you have drank a bunch of vendor coolaid or if you have real world experience, but in my real world experience, there is zero difference in "raid" and non raid drives. As you have said, mostly its just a different firmware. I have only ever experienced this firmware making drives "incompatible" unless they are installed in the NAS or SAN which they were purchased for. I always considered that a big ripoff.
I am however curious, i doubt seek times could possibly be affected by a new firmware (then why wouldn't they do it to all their drives?). Do you have any benchmarks or print/web evidence that "raid" rated drives actually perform better? Like I said, I dont want to doubt you but I have never personally noticed a difference except in price.
As a potential lottery winner, I totally support tax cuts for the wealthy
You cannot violate this rule:
"Pick any two: performance, cost, availability."
That applies to *any* cost. At $100/TB, it's "pick any one". Your average user is just looking for a place to stash his pr0n, so optimizing for cost is perfectly fine.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
Drive space is so cheap per GB these days, for a lot of companies there's almost not much point in RAID 5 anymore (or of course RAID 6 or 0 but whoever used those anyways). If you have an even number of drive slots greater than or equal to 4, mind as well just buy a bunch of large drives and RAID 10 the thing. Certainly there's exceptions and specialized configurations, but in general, as drive space per $ goes up, the storage capacity issue goes down.
Thecus are selling NAS that use Linux software RAID, and it bloddy well suck!
Sure it's fast, but it corrupts the data, I've lost 1TB data, and a friend of mine who bought one also, recently lost about the same. The forums are full of people who lost data, so now Thecus include a disclaimer in their firmware that they are not accountable for any dataloss, how secure does that make you feel?
It can be that Thecus fucked up the embedded Linux they run on it(n5200), but my next NAS is going to be a server pc with internal disks, running ZFS and OpenSolaris or Windows with NTFS.
-H
I just built this last week:
http://www.flickr.com/photos/vancod/3679263377/
8 x 1TB drives: $712
PSU: $92
Case: $29
RAM: $34
CPU/MOBO: $131
Intel RAID card: $275
Cache/Battery: $112
Total: $1385 (all items free shipping)
Don't know how you're going to get $1000 for 10TB and have it be worth a shit.
Even casual involvement excludes total freedom by it's inherent nature. John Valby
From building two or three of these at home myself, my practical experience for someone wanting a monster file server for home, on the cheap, consists of these high/low points:
1. the other poster(s) above are 100% correct about the raid card. to get it all in one card you'll pay as much as 4-5 more hdd's, and that's on the low end for the card. decent dedicated PCI-E raid cards are still in the $300+ range for anything with 8 ports or more.
2. be careful about buying older raid cards. I have 2 16-port and 2 8-port adaptec PCI-X sata raid cards that are useless. why? they only support raid arrays up to 2tb in size. "update the firmware", you say. sure, let me just grab the latest, from 2005, I'm sure that fixes it. oh, wait, my raid cards already have that, and it doesn't remove that limitation. 8 drives, 16 drives, even, and they hard-code a limit of 2tb? lame.
3. I've seen nothing in a home-budget price range that performs as well as linux software raid. My 1.5 yr old 500$ tyan workstation mobo(S5397, in another computer) has dedicated SAS raid that can't seem to do better than 10mbyte/sec throughput. reading data from drives that individually bench out at 50-60mbyte/sec.
4. which leads me to: use linux software raid. It's much more configurable than any hardware raid card, both in supported raid levels and monitoring capabilities. raid disks/arrays can be easily moved from one machine to another, one controller to another, etc. I've moved most of my disks between machines and controllers at least once.
5. I've come to believe over time that what you're really looking for is X SATA ports, not "controller capable of doing raid over X disks". Use SATA "mass storage" cards, or raid cards that will let you use them in pass-through mode to access the individual disks directly in the OS. here you have to be careful you don't get bit by #1, 2, or 3 again, since some raid cards don't behave well when not actually doing raid (I'm still looking at you, Adaptec). this makes it easier and much cheaper, you can mix and match lower-capacity cards to get 8-20+ sata ports for raid.
5.1 "hw vs sw raid tangent" : what happens on a dedicated raid card when you run out of ports? you usually can't span raid cards, unless you get multiple identical fancy (aka expensive) raid controllers from the same manufacturer. all linux needs is hard drives recognizable by the BIOS.
6. when using software raid, buy a decent CPU. You don't need some quad-core beast, but you don't want to be waiting on the CPU to finish your raid calculations. any 2-2.5ghz C2D is probably more than adequate...I've drawn the line with anything under 2ghz.
7. kiss backups good-bye. the price of any decent backup system capable of covering this much storage is WAY over the price of this whole setup. Anything I really don't want to lose gets saved multiple places outside of the raid array, otherwise I factor the potential for data loss as a risk of operating this way. Personally I don't really see how you could do otherwise in a setup like this.
8. be prepared for bottlenecks. you're doing this on a home budget, you probably won't get 300mbyte/sec reads off of your array, no matter how many drives configured at what raid level. I can only get 10-20mbyte/sec across my gigE network going to/from my raid 5 array. This is probably due to the cheap PCI sata cards I'm using. I willingly make this trade-off to obtain the capacity I have for the price I spent.
If any of these points is an overriding concern for your intended use, then you'd have to re-evaluate the importance of all the other considerations.
For me, stability, capacity and price are top three, leading me to research linux-stable cheap sata expansion cards (which is just a nice way of saying, I buy and try probably 2x the # of controllers I actually use, to find ones that won't corrupt data, time out on random drive accesses, or simply not display the real drives to the OS, etc), and compromise by waiting a bit longer for network transfers. Usua
I'd love to hear others feedback on similar personal use ULTRA CHEAP RAID setups.
For software, use OpenFiler.
Da Blog
This article is stupid mainly because it spends over $1000 (something like $1200) on the RAID card, while spending another $1000 on 12 drives. A RAID card that supports 20 drives, not just 12, and mixed SAS and SATA drives instead of just the SATA it needs. Not to mention that the RAID itself can go in SW under the Linux kernel instead of spending on HW to do it. And that single card is a single failurepoint, making the 12x redundancy of the drives kinda irrelevant.
Instead, 4 $25 4-port SATA cards are enough. If you want parallel throughput, go for PCI-e, but just the parallelism from 4 cards in old PCI is enough for most apps. Spend 10*$80=$800 on 10 1TB drives, the $100 on the SATA cards, $50 on a 400W power supply (plenty for 10 drives each pulling about 15W, + motherboard), $20 on a 10-bay case, and blow past $1000 a little by putting a P4/2.4GHz/1GB-ethernet motherboard in there for $50. Now you've got almost 1TB capacity, good thruput. Install Ubuntu server, config your SW RAID, ethernet/webserver and whatever NAS server SW you prefer (sshfs, NFS, whatever). Presto! for just over $1000 (for real), you've got almost 10TB (spend $1100 for the full 10TBs).
The only tricky part might be staggering the drives' spinup kickoffs so the 400W power supply doesn't catch all their load spikes at once. But I'm sure someone can post a bootloader config or patch that will handle the only really wizard part of this whole challenge.
--
make install -not war
I work for a company that you've heard of, and makes storage arrays. The major difference is in between SAS drives and SATA drives, and of course the raid controller.
Big companies see it all... Some jbod like this has seen maybe one or two different environments. Would it work for you? Maybe, but you aren't going to be putting data on it that you're making money from. If you are, I suggest you get your head checked. Large companies, have relationships with the hard drive manufacts and we get to cherry pick our hard drives... basically, we pay A LOT more for drives that have much higher tolerances - they might be the same model number as something you can buy from newegg, but when you have 15 drives or more in a raid-5, I would hope you have the brain to spend some cash on good hard drives... if not, a spinal cord would suffice for you.
Tom's hardware's blatant lying aside (12 disks + controller >> $1000). Its a good example of what people are getting sick to death of in the server market. i.e. add "enterprise" to a name and everything gets needlessly expensive.
if you look around at a couple of fibre-connect (or even scsi-connect) jbod's the cost is ludicrous even without the drives. Some of that is due to the weighty licensing costs of fibre channel and some of it is just greed (and an inability to efficiently produce components).
On the flip side, my personal experience with hardware raid controllers has never been great, the controller dies and you no longer have anyway of accessing your data cause only that controller knows how to read the meta data on your disks. Its also rather pointless in most situations these days. What exactly are you going to do with the much disk bandwidth? did you get a 10gb network suddenly you can share it over? About to do some film-quality HD video editing (in real time) for harry potter?
With the speed of most harddrives these days, theres just very little point using a hardware controller to build a "cheap system" when you could use a software one thats so much easier to replace if it fails. Your still going to get the speedy access.
Last time I checked, ZFS was freely available in either OpenSolaris or FreeBSD. No cost.
Now, you may not like the hardware compatibility guide (I don't), but if you use JBOD controllers and standard ethernet devices, it should work fine.
I have a lot of spare drives (250/300/500) from my web hosting business because I rotate them out every couple of years. I found a 16-bay 4u case on ebay for chump change, grabbed some cheap Hipoint cards, and set to work building a backup storage array using software raid. As it turns out, the case was cheap because some of the caddy connections were sketchy, and of course the cheap cards would randomly "not see" a drive now and then. After spending way too much time troubleshooting and configuring things, I finally got it all set up and working. Had 4 TB of backup storage. Unfortunately, I never bothered to MEASURE the damned case, which turned out to be 26" deep. This conflicted somewhat with my 24" rack, especially when attempting to close the rear door.
Plan B: Garage sale with a 16-bay case and dozens of hard drives available. A D-link 321 NAS with built-in web server (for config), ftp, telnet, a 1gbps ethernet port, and a pair of WD 2 TB drives in RAID0. Roughly $500 for 4TB. It's barely larger than the 2 drives and I have it stashed in another building on the far end of my property. Honestly, DIY just isn't worth is sometimes.
Nothing worthwhile ever happens before noon
started out with just over 2.5 Gigs with 300 Gig drives and now extends to over 10TB with 1.5TByte drives all done with RAID 5 and no spares
Currently have a half dozen of them in the house with various loads of video from "live streaming eagle nest cameras" and there are several more at the biologist's and out in the field.
Started out about $2500 (Canadian) and are down to around $1500 - but I've started putting faster chips (than the original Celerons) because I found that having them online was a good thing when doing video editing as a render farm but far better when they had some power :)
One thing to remember is that the bit error rate on these large drives is "per megabyte" which means the larger the drive, the more likely there will be a failure. I've been bitten once by a double failure before I could get a spare in and integrated - lost a whole array. I've seen one study that shows that the likelihood of a second drive failure before a spare is integrated, even if spare integration starts immediately an error is detected, is almost 100% once we hit about 3-4TBytes/drive. I've started using the drives as mirrored pairs and spreading the load (of video files) over them with other means than RAID 5 - even RAID 20 is not going to be enough IMHO - need something new like a completely new subsystem concept I saw a note about a while back but can't find at the moment - I'll post more if I recall/find it.
Been there, done that, paid for the T-shirt
and didn't get it
Or, if you have a somewhat modern Intel-chipset board (with 6 SATA ports, and Intel's RAID-5 capable "Matrix RAID",) you could buy 5 or 6 Seagate or Western Digital 2 TB drives for $230 each. No fancy total-price-doubling adapter needed.
If you're willing to risk your data, 5 drives is enough to do a 10 TB RAID-0; if you're a little less willing to risk your data, and have a motherboard with either a PATA controller, or an additional SATA controller for your optical drive, or are willing to live with an external optical drive, you can go whole-hog and get 6 in a 10 TB RAID-5.
Another non-functioning site was "uncertainty.microsoft.com."
The purpose of that site was not known.
Personally, I'd rather like (if I had the cash,) two SSD's in RAID-0 for boot, one optical drive, and three 2 TB drives in RAID-5 for important data, if I was going for an all-in-one high-speed, high-capacity desktop.
Instead, I have my home server with its 8 drive bays. (Currently only have four drives, in two arrays, totaling 2.5 TB usable space.)
Another non-functioning site was "uncertainty.microsoft.com."
The purpose of that site was not known.
If you want to see real drive arrays snag yourself a pass to the RSNA show where medical imaging technology vendors strut their stuff. They consume massive amounts of data and it's typical for storage products there to have hundreds of drives in them and be the size of a London apartment.
But this really isn't new and I can't get too excited about this project. I have a Mylex SCSI raid card on my desk (actually I have a box of them) worth all of about $12 on ebay tehse days (never mind it's still $200 from dealers, or was $2200 6 years ago) that has 3 connectors and will manage up to 45 drives (15 per scsi bus).
Show me a big array of 2.5" notebook drives on a desktop or a raid array of SSD drives and you'll raise my eyebrow, but I've hooked up 10 drives before: you stick them in a cabinet, attach the cable, configure the array then go. This is not really a big deal.
Need Mercedes parts ?
You'd have to be really pervert to need 10 TBs of mass storage. That's a lot of porn.
6 x 1.5 TB Segate = 720 USD Motherboard Intel (6 SATA ports) = 70 USD Processor Core2duo - 100 USD Ram 1G: 10 USD Case + extra power supply: 50 USD Flash drive(to hold os): 10 USD FreeNAS(provides software RAID): $0 USD 9 TB = 960 USD, all included :-)
I've been using the 1680 series for a couple of years now and they've been rock solid, for the most part. I had one that was delivered bad, replaced it and the replacement is running smoothly a year later. Have you got any kind of outside numbers that show them having a higher failure rate / data corruption rate? The brand I've had problems with in the past has mostly been adaptec.
--- It is not the things we do which we regret the most, but the things which we don't do.
I suppose I hadn't thought to consider the abominable state of most ISPs these days. For my usage model (a ton of static data with occasional additions), it might work if I sync'd it locally before switching to remote traffic, but I suppose that, in the end, it really depends on how draconian your ISP is.
Laws do not persuade just because they threaten. --Seneca
My understanding is that they test all the drives, and pick the ones with the tightest tolerance to brand as "RAID edition."
You can draw your own conclusion as to the value of that.
When I first saw this article on Tom's Hardware I just about laughed my ass off. 10 hard drives in a RAID0 they propose? Okay, that is sick as fuck and everyone knows it. Must be a really slow summer. Anyhow, here is my current party piece. Prices in Canadian loonies: Solaris Express snv_114 [free] Intel C2D @ 2.66GHz, 6GB DDR-II, Asus P5Q-E, plus trimmings and 2x Seagate Barracuda 7200.11 1TB internal (ZFS boot+mirror) [Only 'new' part was the mobo, $200. Total value is probably like $800] 2x LSI SAS3442E-R PCI Express HBA, these were relatively cheap. [$230 each] 2x AMCC/3Ware CBL-SFF8088IB-20M 2m (SFF-8088) to (SFF-8470) Cable [$80 each] EnhanceBOX E8 MS External Mini SAS Enclosure with dual PS [$800] 8x Seagate Barracuda 7200.11 1500GB 32mb 7200rpm, in 2x RAIDZ [$140 each] So after tax like $3500 Canadian. It is worth it though. A *little* bit of redundancy to protect your data at the very least is a good idea. RAID1+n is supposed to be best but, my data is largely multimedia. I might be miffed to lose files but I certainly won't be out of a job. I can upgrade RAID size by swapping out hard drives for bigger ones (must be done in block of four. What comes after 2TB drives I wonder?). The external enclosure has 2 RAIDZ's consisting of four drives each. If a drive dies, I hot swap it out. The odds of a drive dying before I replace it with a bigger one is low. If a controller dies, I can switch back and forth between RAID sets until I find a replacement controller. New controller type doesn't matter, as long as its mini-SAS and supported by Open Solaris! Battery backed controller doesn't matter since ZFS is always consistent on disk. Redundant power in the external enclosure. If the PC dies I can just replace it (needs 2x standard PCI-e x16 slots though, which not all mobos have.) All in all I think its a pretty sweet setup. I get 50+MBps copying files on or off of the system. Only drawback is I had to upgrade the firmware on some of my drives because of Seagate's 'bug'.