Slashdot Mirror
Terabyte Storage Solutions?
DeMechman asks: "As many on Slashdot may know, storage is one thing which you can never have enough of. Given the current situation with CD/DVD rot (Personally I can attest to a 10% attrition rate) hard drives in a RAID configuration seem to be a better and more economical solution. If you own more than fifty CD/DVDs, it can be a daunting task to find a file. I am wondering if anyone has found a hardware solution that can inexpensively be set up to handle 10 or more 250GB HDDs in a RAID configuration. Primarily, has any case manufacturer tackled this niche market yet?"
I'd say that $2.82/GB, for a well-built, well-designed 14-drive 3U RAID (0, 1, 3, 5, 0+1, 10, 30, 50) hardware cabinet with dual-2Gb/s fibre channel connectivity, dual-100mbit ethernet and serial for monitoring and management, excellent Java setup, management, and montoring software, redundant hot-swappable power supplies and fans, and that works and is qualified for use with Windows, Linux, and Mac OS X, qualifies as "inexpensively". But that's just me.
http://www.apple.com/xserve/raid/
Academic prices for:
1.00TB - $5399
1.75TB - $6749
3.50TB - $9899
It's not RAID, but you could buy a 1-terabyte drive from LaCie.
In soviet russia, You ask not what country do for you, but what you do for country!
Oh wait...
Apple is one of the cheapest, at 6000$ (with drives)
See page here.
Comment removed based on user account deletion
http://www.intel.com/support/motherboards/server/c hassis/sc5200/index.htm
Just bought one myself.
You can get em at:
http://www.bellcomputer.com
Let em know G Force Hosting sent ya!
I have a TB here, and rather than raid, I decided to do a nightly "rsync" mirror to a "yesterday" partition.
The two advantages of the nightly rsync over RAID are
- It protects against user-error too. If I make a bad edit, I can always 'diff' against
/yesterday/home/me/...'
- It makes upgrades of both hardware and software easy. Since my live backups are excactly that (live, and tested every day), one machine can be fully upgraded while the other acts as the primary one for a while.
Important data also gets backed up to another large HD in my car and DVDs in a safe occasionally, to protect against a fire or burglars.you can "cheaply" buy 3U rack mount cases that hold 15 drives in hotswappable SATA or SCSI cages up front. Combined with a 3ware 9500-12, and leave 3 cages empty(or spare drives just not cabled up), this will give you 2.75 TB in each unit of raid5 storage. If you were really hard up for space, you could use a pair of 9500-8's and this would give you 3.25 TB per unit. Some 4U units hold 16 drives, which gives you the full 3.5TB in 2 x raid5 arrays.
I have 8 x 160GB Maxtor drives in a RAID5 array. It's fast, relatively inexpensive [Fry's Electronics recently was selling the 160s for $69/ea]
/dev/md2 1.0T 521G 522G 50% /ext
The 160GB drives used to come with a Maxtor [Promise] ATA-133 card. Two of those will support eight drives. Not the most optimal arrangement because of the bus having two drives on each channel, but it doesn't seem to affect performance too much since it is striping the data across all of the drives. I'm assuming it stripes in order, so you'd want to stagger the drives such that 1 & 2, 3 & 4 are not on the same controller.
Output of df -h:
The cost to assemble something like this?
~ $600.00
8 x $70 for the 160GB drives
2 x $20 ATA-133 controllers
The biggest issue is that there is no easy way to back up the array. You could use RAID 6 and have two drives worth of parity info, but it still leaves you vulnerable to a catastrophic hardware (or building) failure.
Anyone have any ideas on how to back up 1TB in a home environment? i.e., not $3000 tape drives & $200 tapes
Good IDE hardware RAID controllers with Open Source drivers. Appears as a single SCSI drive to Linux. We swear by them.
My God! It's full of Voids!
We just added a couple of these at the office. We used a SATA RAID card from LSI Logic (formerly AMI MegaRAID) and on top of the 6-port device added six 200GB Western Digital drives. From that page, a 200GB Maxtor can be had for around $85.00. Add in a 2U case, which is probably the most expensive part at around $300.00, and you have yourself the most expensive components of what you need, subtract the motherboard, processor, and all that jazz (which can be had for another $300.00 or so). Running Linux LVM with Samba-3 and Winbind for full Active Directory integration and authentication on top of an ACL-enabled ext3 filesystem, of course! ;)
I bought a case from http://www.servercase.com/, a 3Ware RAID Controller and 8 200GB IDE drives. I've got 1400GB of usable space in RAID5. It runs Linux with Samba and NFS. I also use it for a MythTV Backend.
Unfortunatly, once you have all this space, you WILL find a way to use it all and need more. I put this system together about 10 months ago, and it's at 85% capacity now. I'm preparing to build a new server with 12 250GB drives, to have just over 4TB between the 2 systems.
www.raidweb.com Bought one of these at my previous employer and we really liked it.
I use a Hard Drive Enclosure for backing up files. With IDE HDD's getting less and less expensive, picking one of these versatile enclosures up for less than $50 is a good value. I own a DVD burner but rarely use it for data storage since the enclosure is way more convenient. Now as far as 10 250GB drives in a Raid configuration, how redundant redundant do you need you data to be? Or is it that you're just overly cautious after having your backup DVD's fail? Just curious.
Yes, I've used an HP/Compaq DLT auto-changer that will do the job.. Don't remember the price offhand, but I remember it was in the over-$100k range.
We use Linux LVM to take snapshots and then do a hot backup of that data to an archive box. That archive box contains removable hard drives (tape drives are just crap), and we then take the pysical drives to an off-site location to provide security and all the goodness that comes with off-site storage. We also use rsync to synchronize our production NAS devices with a parallel NAS device, to which we can hot-cut and have a current copy of all our data to a 15 minute window. Because rsync (with ext3 ACL support, mind you) only copies what has changed on the filesystem, it goes relatively quickly. You can find my rsync packages at ftp://bagel.express.org/ (as well as patched Samba-3 packages that really work with Winbind and some updated kernel packages for LVM+snapshot support) at that FTP site.
I can answer your question, as I've just built one as a giant backup solution for our hosting company.
:)
I went with Serial ATA for a couple reasons:
1) It's cheaper and has more capacity than SCSI;
2) Cabling is not a mess as it is with regular IDE (if you've never seen serial ATA cables, the first thing you will notice is that they are small!);
3) It can hotswap, unlike regular IDE;
4) It's not that much more expensive than regular IDE.
I custom-built a 3U server from InterProMicro. They are a small (local if you are in the Bay Area) SuperMicro reseller that does great work. (If you need something, call and ask for Andy. Tell him Erica from Simpli sent you!)
The machine I specced out was as follows:
* 3U case with 8 hot-swap SATA drive bays;
* 8-port 3Ware 8506-8 SATA RAID controller;
* 5x250GB SATA drives in a RAID-5 array;
* Dual Xeon processors.
The 5 drives give you 1TB of storage, and expanding up to 8 gives you 1.75TB. I would also recommend a separate mirrored SATA 10KRPM array for the OS if you want really fast speeds.
This whole solution (Xeons; 5 drives; 3U case) cost just over $3000... which is pretty reasonable for 1TB of network-accessible storage. Interpro has solutions that go up to 24 SATA drives, which at 250GB each gives you an ungodly amount of space (5.75TB, if my calculations are correct.)
My suggestion is to go with a niche server builder like InterproMicro over Dell or Compaq or any of those guys. You can get the same high quality from a custom manufacturer without paying the steep brand name price from a larger manufacturer. As for the drives, any time the goal is "as much space as possible", SATA should be your first choice.
Good luck!
Simpli - Your source for San Jose dedicated servers and colocation!
It's almost funny... Once you start talking about RAID with more than 2 drives, IDE is at a disadvantage.
I'm not referring to performance, reliability, etc. (although those are serious issues), but about price.
If you have a master & a slave, then you reduce performance... That can be a very serious if you have a RAID configuration. So, if you want to put 7200RPM hard drives together, you start to need a 6 or more channel RAID card (whereas a single channel SCSI RAID card would work fine). And guess what? Decent quality 6+ channel RAID cards are very expensive, perhaps even negating the savings from using IDE drives rather than SCSI in the first place.
Remember, that's based on price-only... I haven't even begun talking about how much worse the performance would be, or reliability issues with using inexpensive IDE drives.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
-What RAID level you want (5 usually requires better hardware)
-Whether you want hardware RAID (I strongly recommend this) or soft RAID
-How much redundancy you need (Battery backup cache? Redundant controllers? Hardware environmental controls?)
If you are looking for good pci cards, I would strongly suggest a card from 3ware, and a card from a place such a Seagate. Getting a super-duper cheap card when terabytes of data are on the line is just fundamentally stupid. You can save some bucks now, but be ready with your next Ask Slashdot: "How do I recover data from my dead RAID?" Seagate now has a nice 5 year warranty, which match well with good quality and reasonably cheap drives. Look at some of the SATA drives like the Barracuda. However, any decent quality drive maker can work. If you have even more money, you can look at some of the things offered by places like StorCase. A larger initial investment can become cheaper as you scale up the cheap harddrive count, and it can be a good thing in the long run. Obviously, the more time you are willing to invest doing things yourself, the cheaper you can get to some extent vs premade items. However, no support as well.
Do read up on some of the fundamentals of RAID: Everything you need to know (and lots you don't) is probably at least mentioned in the PC Guide on RAID. Look through that. Things like hot swap and hot spares are important to understand. Finally, you should remember to check compatability. Unfortunately, I for instance have not been able to find much of anything in the way of controller cards that is compatable with OS X (except the obvious, the XServe RAID). So I have something set up on a BSD box in my server closet that I then link to, more like a storage appliance. Happily, the 3ware cards and many others are now compatable with a wide variety of *nix and BSD flavors along Windows, but do check to make sure.
Last but not least, remember this!: RAID is *not* a backup solution, but an highly redundant onsite storage system. Have another form of backups, even if it is just a RAID 1 off site, or DVD-Rs, or something. If a disaster happens (thieves, fire, nuclear destruction, John Ashcroft) on site storage won't save you.
Promise has a nice off-the-shelf solution and you can get it for arround $3600.
If I were going to do it I'd build it my own by combining a nice case and a 12 port 3Ware controller with whatever server configuration and SATA drives I wanted to get.
set softtabstop=4 shiftwidth=4 expandtab nocp worlddomination
Only comes with a 1 year warranty? I'm sorry, but if I'm gonna be spending over $1,000 for a storage solution, it better come with a 3 or 5 year warranty at the least. Heck all of the new Seagate drives come with 5 years warranty!
eTrade SUCKS
That's one thing that I always appreciate product builders keeping an eye on - for example, the 3.5tb XServe RAID, while more expensive (and providing more features), specifies maximum heat output of 1365 btu for the disk array and 990 for the server (assuming all 17 disks running full tilt with both G5s pegged). Not bad for a 4.25tb system. Under lower load, heat output from both drops substantially.
You're special forces then? That's great! I just love your olympics!
10% attrition? In my case, I can't remember a CD-R I recorded that ever failed (I don't use CD-RW, maybe these are something else...). All my source backup are on CD-R, and I make quite a lot of them. I also burn a *lot* of music compilations for my car. Some of the CD-R there stopped working after (quite) a while, but it is only because a car is a harsh environment for a CD-R.
The CD-R brand must have something to do with it. I only use Sony's CD-R. Not for a particular reason. Only that none of them ever failed me.
Thus, my opinion is that CD-R are one of the bests (if not the best) solutions for non-industrial backups. By industrial, I mean freaking mission-critial multi-GB multi-millions dollars worth backups.
Note: I never tried DVD-R. You must code a *lot* of lines to make your projects' sources weight more than 700 mb. (Hum, quick cvs tree check: 75 mb... ok, I might be wrong here. However, these 75 mb were a *lot* of work for me...)
perception is reality
- Find any tall beige-box case. ($150)
- Find 9 good 250g Serial ATA drives. ($100 each = $900)
- Get an 8-port serial ATA hardware RAID controller like these ($300)
- Get a good 400-500W power supply ($200)
- Any motherboard and CPU will do ($200)
- Spend a few extra bucks on gigabit ethernet ($50)
Put 8 of the hard drives into a RAID-5 array. (1 for your O.S/system use). That makes about 1.4 TB for only $1800 total. The 3Ware IDE raid thing works great with FreeBSD, which is what we use for everything.Rip all your CDs as FLAC so that (1) you never have to rip them again (it's lossless), but (2) it's half the size of saving WAV files
At least that's what we've done with our 68,000 CDs we have here.
I have two systems, with about 1.3 and 2.5 TB respectively for archiving DVD quality video and MP3s. I looked at RAID but found it was not necessary. I prefer to manage the disks (some are removable) and do not need high performance even when streaming the video over my in home LAN.
I use DVArchive with DVD or satellite to ReplayTV for video capture and play back, DVA is great for managing multiple volumes and dynamically discovers vidoes if I want to move them to another drive. It also supports copy/move between the two systems (I use a 1Gb switch between systems). CPU performance is not key for play back though it is critical for transcoding (I use a dual processor system for transcoding and it smokes my single CPU system).
I have a LARGE MP3 collection (forgive me for not publically admitting to its size) and I find the same systems/drives are ample for supporitng a digital audio library. I switched to iTunes for managing music (MusicMatch melts down when the number of files gets large) and stream it with SlimServer to squeezebox devices for high quality playback on home theater and other receivers.
My recommendation is to go with generic disk drives - brand names, 7200 RPM with 1-3 year warranties --I get them locally on sale for under $150, sometimes $130/250GB, thats 52 cents per GB, a little more per GB than a DVD-R disk but more reliable and infinitely more flexible. I can recreate a DVD off of the disk image if needed.
I am more concerned with heat and power consumption (it adds up) than disk performance, someone will need to explain to me why I'd need to mess with RAID for this...
Next, I'm using a 3ware 7810, which is an 8 port PATA/100 RAID controller. I'm currently using 7 ports with 120GiB Seagate drives for 3/4 a terrabyte of storage. (I want to put a hot-spare on the 8th port when I can afford it.)
Including a beefy Antec power supply, the case, the drives, and the RAID controller, it all comes in in the neighborhood of $1000. (Don't forget to add in mainboard, CPU, RAM, Network, etc. in your calculations.)
Overall, I'm extremely happy, although being as how it's PATA it's a bit cramped in their with all the cables, but I've done some work ensuring there's no excessive amounts of ribbons around the case. Likewise, I've mounted thermally controlled fans on exhaust and high-range stealth fans on intake everywhere so things keep reasonably cool. However, while the Seagates I chose for low noise and despite adding some sound dampening as well, it's still not a quiet box.
The cheapest way I know is still a large PC tower case, a 3ware SATA raid controller a big PSU and a couple of large fans withs attitude. On the PCI side you don't need much unless you want to do gigabit (or of course just shove your server in the same case and dont do the I/O networked)
I never knew this, and apparently many others didn't either, but if you use hardware RAID the disks are tied to that card.
m l#28
More info here, plus the ever-acidic jwz calling people dumbasses, dipshits, and more fun!
http://jwz.livejournal.com/368307.html
http://www.dnalounge.com/backstage/log/2004/07.ht
John Kerry is a Joke!
We (The Binghamton University Computer Science Department) employ 2 debian raid servers. They make use of a 3ware ATA 12-port card and their (3ware's) hot-swap enclosures (whoever said hot-swapping with ATA is not possible is incorrect, we do it).
It uses a 9 external 5.25 bay case (enlight) with an Antec 550W power supply to handle the 12 drives (plus a seagate system drive in the internal 3.5" bay). This has worked very well.
We use Maxtor 300GB drives in one machine (RAID55) and have lost 5 of 20 drives we purchased in 6 months. The other uses Western Digital 200GB (RAID5), and we've lost 1 of 12 in a year. Manufacturer DOES matter. WD replaced our drive in days, Maxtor makes you jump through hoops and tries to deny the problem for a while, just to finally decide to replace the drive, then take 5-7 mroe days to get it to you.
All in all, these machines cost us under 7K each and perform very well. However, if I bought one today, I'd get 3ware's SATA card and Seagate's new 400GB SATA drives instead. Whoever said ATA cables are a pain was NOT wrong, and these drives would give much better performance.
You can RAID USB floppy drives or keychain drives with OSX. Might take a few keychain drives to get up to 1TB, though
I drank what? -- Socrates
I don't understand why people automatically assume that DVD+-R/RWs are prone to the same problem. The recordable layer in these disks is buried in plastic. It's NOT on the surface. There's no oxygen coming to it, so in theory DVD+-R/RWs should be a heck of a lot less prone to "rot".
And i've installed quite a bit of these:
* SuperMicro motherboard (any of the newer ones, depend on your choice of architecture). Be sure to get one with PCI 133/64 and gigabit onboard.
* 3Ware RAID board(s).
* Chembro rackmount cases (they have a very nice one with 16 SATA hotplug slots with backplane and all)
* Don't go cheap on the power supply. You'll need at least 600W. I always go for redundant ones.
* 16 SATA disks of your choice (250, 120 or 80GB)
* Linux!!! (Be careful with fedora core2, it doesnt support nativelly the 3Ware cards - you'll need to compile your own)
Of course you could save about $1000 by using a cheap motherboard, chassis and PS. But it really pays off using the good brands on those.
By the way, you should always get an extra hard drive (or two). They will fail (sooner or later) and you don't want to be left hanging.
I was just researching this, I myself am making a 1 TB storage server. I have come up with the following solution. 1 Broadcom BC4852 Serial ATA Raid drive. Now this thing works wonders, and only costs (approx) 362$. It can change raid levels without bringing it down (IE, start with 1 HD, pop another in once you buy it, SATA is hotswap, and Move to raid 1 or Raid 0 automagically, pop another in and go raid 5.), it supports 8 drives PER controller, and you can use 4 controllers (AND THEY ALL ACT AS ONE). This means you can have up to 8 TERRABYTES using 250 GB drives, or 7.75 TB raid 5. The 160 GB drives are about 169$ a piece so add a basic motherboard and chassis and you got a full system. Dont forget a bunch of drive trays if you want to hot swap.
If you want to spend the extra money and have a warranty and fancier case, look at Nexsan , or EMC's AX100. Scary that EMC is selling something cheaper than the competition, but they are. Sorta disturbs the natural order of the universe. Still, either will set you back several thousand. The AX100 looks pretty impressive on paper. Options for dual controllers, and up to 3 TB in a 2U space. Haven't tried one myself yet.
Disclaimer: I work for a storage integrator, both are brands we sell.
Ignorance is the root of all evil.
I'm in the middle of doing this myself... I'm saving the money to buy the drives I need. I've already got the server and array below.
;-)
2x Sun Ultra 10 desktop machines (360mhz / 512mb / 2x 18gb drive (hot-pluggable drives)) @ 150.00/ea (EBay)
2x 3' HVD cables @ 28.00/ea
2x X6541A Sun Dual Differential Ultra/Wide SCSI @ 100.00/ea (EBay)
1x Sun StorEdge A1000 storage array @ 120.00 (EBay)
10x Sun Ultra2 SCSI Drive Sleds @ 58.00 (EBay)
7x Seagate - ST1181677LCV 188gb Ultra2 SCSI drives @ 550.00/ea (PriceWatch)
Total for 1,128gb of Raid-5 storage: $4584.00
The trick is, with this setup you will have two machines with redundant access to the drives and data in the array. The Ultra10 is enough to handle any home use I can think of, and paired with Solaris 9 or even Linux will be blazingly fast. I just think that it's more expensive than any comparable SATA setup... great for us Sparc lovers tho!
My school asked for me to create just a solution. Here's what I ended up with:
* 4U Enlight Case.
* 2 5-drive SuperMicro Hot Swap removable Drive cages. (SATA)
* 3Ware 12 port 8506.
* Supermicro Dual Xeon Motherboard.
* 2 Xeon 2.4Gig Hyperthreading Proccessors.
* 10 250 Gig SATA Harddrives.
The 3Ware card has a limitation of 2 Terabytes for a single volume, so we used a 9-Drive Raid 5 with 1 hot spare to make our large drive.
We used Debian Sarge, and BackupPC to backup our school's servers. We can backup EVERYTHIHNG now.
Oh, and this solution cost us less than half of the pre-designed solutions. The school has been very happy with it indeed.
Or, if you want really durable read-only storage (i.e. lasting a few hundred years without maintenance), you could use the little 1x1 LEGO blocks as bits.
Therefore, a mere eight-by-eight city block area could store a full 1 terabyte of LEGO-ROM, with no worrying about DVD rot or head crashes (although access speeds would leave something to be desired).
>;k
This solution looks very interesting to me.
http://www.areca.us/IDERAID.htm
It takes up 3 external 5.25" bays and allows you to connect 5 3.5" drives. It provides expandable RAID 5, all internally with it's hardware and simply looks like an ATA or SATA device to the computer.
Has anyone here actually used one?
kiwi
--
System Architecture
Toshiba TMPR4927ATB 200MHz 64-bit RISC processor
64MB on-board cache memory with ECC protection
Areca 5 channels IDE controller (ARC600-66) with enhanced H/W XOR engine
NVRAM for RAID configuration & transaction log
Write-through or write-back cache support
Firmware in Flash ROM for easy upgrades
RAID Features
RAID level 0, 1 (0+1), 3, 5 and JBOD
Multiple RAID selection
Array roaming
Online RAID level/ stripe size migration
Online RAID capacity expansion and RAID level migration simultaneously
Automatically and transparently rebuilds hot spare drives
Hot swap new drives without taking the system down
Instant availability and background initialization
Automatic drive insertion / removal detection and rebuilding
Disk Bus Interface
Ultra ATA/133 compatible
5 channels, operating in parallel
5 hot-swap drive trays
48-bit LBA support allows disk exceeding 137GB
Staggering the Spin-Up of Individual Disk to Solve the Power-on Surge
Host Bus Interface
ARC-5010
Dual ATA interface-Ultra ATA/133 & Serial ATA 1.0
Ultra ATA/133 compatible Transfer rate up to 133MB/sec
Serial ATA 1.0 - 1.5Gbps(150 MB/sec)
ARC-6010
Ultra 160-Wide LVD SCSI; Transfer rate up to 160MB/sec
Tagged Command Queuing
Concurrent I/O commands
Um... ever consider the mind-bogglingly simple solution of:
ls -R> ~/dvd.index/<disc_label> for each dvd
grep "<whatever_youre_looking_for>" ~/dvd.index/*
So for that 50 TiB total, you need 50/1.4 = 36 systems. 36 systems * 8 drives = 288 SATA drives spinning. How often do you have to replace one? I'm just wondering as I have had 4 x 200 GB drives in RAID 5 in my personal system for just under a year now and I've already had to replace one. Didn't lose anything, and it was under warranty, but in a month, I'll be out of WD's crappy one-year warranty and I'll have to start buying drives as they fail to keep my data.
Karma: pi (Mostly due to circular reasoning in posts).
Instead of looking at a semi-commodity 1TB solution - which is a PITA for needing an industrial strength case, power supply, drive controller card and HVAC, you need to look at the other end:
.8TB.
Two or three fairly normal PCs with STANDARD drive controllers, PSUs and HVAC.
Look, we're talking file servers here. 128MB RAM is gobs if you aren't running any other service on 'em. Pick and OS, any OS: 2000 gets you dfs, *nix gives you NFS. Both give you a homogenous networked file system.
So...
Standard case/PSU/cheapo CPU (Athlon mobile or Via or P3, for lower power consumption)/RAM - That's $250, maybe. Add another $20 for a gigabit NIC or two per machine.
4x 200GB drives @ $110 apiece (pricewatch shows $96 as the low price, but I'll go $110 for a little wiggle room)
So... something around $700 gets you
Buy three machines. $2100 gets you tons of storage and scads of redundancy no matter how you look at it.
This is the philosophy I use in setting up my file servers (now serving 6.5TB!). Over time I've added 3ware cards, upgraded PSUs and added gobs of RAM, but my basic starting point is a very modestly-appointed system.
-- I wanna decide who lives and who dies - Crow T. Robot, MST3K
Could you tell me where do you find good 250g SATA drives for $100 each, please?
"The number of Unix installations has grown to ten, with more expected." (Unix Programmer's Manual, 2nd ed.; june 1972)