Slashdot Mirror

Sun's service has been sliding for some time now. Oracle appears to be accelerating that decline. We had some RAIDs, originally purchased from StorageTek before the Sun acquisition, come off of the three year warranty they were purchased with. We've been unable to get Sun (now Oracle) to recognize the RAID's serial numbers to get them on the maintenance contract for quite some time now. You'd think Oracle would want our money?

Not for kit that old. They'd prolly rather sell you a new array, which their people can actually support. If they even have spare parts for it they probably can't locate them. How many years ago did Sun acquire StorageTek? 5? How old is that pre-acquisition array of yours? At least 5+ years old. Given its age and limited capacity/performance, may I ask why you're still using it? And given the plethora of quality cheap storage arrays on the market, such as Nexsan http://www.nexsan.com/ why would you not just replace that StorageTek array with such a unit instead of continuing your masochist ways with "lock in" vendors such as Sun and Oracle?

1. Software RAID - does well in benchmarks, but it does rely on the CPU. This has gotten better with the advent of multi-core and especially multi-cpu + multi-core, but the other thing to keep in mind is that the OS IS doing other things. What this means is that an errant program does have a chance of affecting your RAID. Could be absolutely disastrous. RAID (true RAID, not RAID0) only works if the RAID manager is doing what it is supposed to do. I wouldn't rely on OS software RAID. There are too many variables. Also, things like hot swapping might not be available. A RAID where you have to bring the system down to do a harddrive replace isn't all that beneficial IMHO.
2. Hardware RAID card - This works in conjunction with an OS driver. Combined with hotswap capable backplanes, this can be a pretty good solution. The problem with HW RAID is that the processing power of the cards is often times week. Upgrading to a faster processor sometimes isn't possible without compromising the entire RAID set (esp. true if changing vendors). End user machines (until PCIe) were somewhat bandwidth limited. 33Mhz PCI, which is what most (somewhat older) desktop machines have is not enough oompf for a large RAID. Linux driver support can be spotty... and buggy. LSI, for example, has made some interesting RAID products, and while Linux users enjoy using older equipment, LSI pretty much doesn't support some of their older cards anymore. Granted, those older cards (which were top of the line 5 years ago), probably aren't that great today, but the fact that they are losing their driver support is frustrating.
3. JBOD - Well.... not much to say here... there isn't much benefit. However, realize that if you can separate disk pathways AND you layout your disks to use each drive efficiently (e.g. /usr/bin on one disk, /usr/lib on another, etc.) you can get some pretty dramatic performance gains, even without RAID striping.
4. RAID subsystem - This is the one not specifically mentioned and it's what I recommend. The benefits of an external RAID subsystem is the flexibility. These units can often times be shared across multiple machines (and NO I'm not talking NAS... I'm talking direct attached SCSI, iSCSI or Fibre channel), can support large amounts of cache and can offer good RAID levels. Since the device manages it's own space, the device itself can notify you of hardware failures and such with relying on coordination with a proprietary driver. Since the computer device just sees a normal drive, there are no driver issues. I can completely saturate (easily) a 2Gbps fibre with one of these (with a relatively small array). There a several manufacturers out there that use inexpensive SATA drives... but there are SCSI and fibre drive based units out there too. I like Nexsan, http://www.nexsan.com/ probably the most. I have also used VERY inexpensive Arena based units, http://www.maxtronic.com./ Both work really well. Pretty easy to get 100MB/s to 200MB/s with these devices and still have almost equally as fast writes while using RAID 5 or RAID 6. If you are looking for a cheap mirroring INTERNAL subsystem, I have used Accordance ARAID systems. I recommend those as well if you just need a two drive mirror. Obviously you get more of performance hit.... but good if you want OS independent HW subsystem RAID that is internal, http://sewelldirect.com/araid-2000-sata-raid-1.asp ?source=froogle&utm_source=Froogle&utm_medium=cse.

I can pretty much guarantee that once you've switched to a HW RAID subsystem, you'll probably never use anything else. The extra money spent on those is well worth it (IMHO). To keep costs down, ebay is obviously your friend. I've purchased 4 drive Arena units for less than $400. I

Or ... you could buy a Nexsan SATABeast and get 42 drives and 20 TiB of storage in 4u and not wait a year. No real story here.

or here:

http://www.nexsan.com/products/products/satabeast/ satabeast.html

(only 42 drives. but it's a 4U and the drives appear to be more evenly spaced than in the generic 5u rackmount.)

Sun will probably not provide drive sleds for empty drive slots.

It's ridiculous to have to buy the exact same Seagate part for 2x or 3x more. No Sun warranty is going to replace lost data. Besides, the whole point of ZFS is to provide fault tolerance and recovery.

Also, the same storage concept has been on the market for over a year, albeit with 42 drives:
http://www.nexsan.com/products/ATAbeastlow.pdf

Do not use XServe RAID. It's the worst possible pseudo-enterprise SAN product. This is not to rag on Apple in general - the company is full of smart people, many of whom are friends. This is just a lame product in an otherwise excellent product line. There are plenty of SATA based SAN storage devices out there which are cheap. I'm partial to Nexsan, having worked with them, and if you need slightly higher quality the Sun Storagetek, EMC/Dell boxes, etc. Software RAID (Veritas or open source) striping on top of large HW RAID (RAID 5, or RAID 10) SAN storage array stacks works just fine.

Oil companies which do a lot of Pre-Stack processing, i.e. raw seismic data, need an awful lot of disk storage: We're currently in the ~50 TB (geographically mirrored RAID5 servers) range, and this is with Post-Stack only.

Going to Pre-Stack will generate 10 to 100 times as much data, which means that 500 TB to multi-PB is where we'll be in a couple of years. Having 750 GB SATA drives in a Nexsan SATABeast http://nexsan.com/products/products/satabeast/sata beast.html enclosure results in about 27-28 TB of usable disk space in a single 4U rack unit.

Very nice!

Terje

We just got one of these: http://www.nexsan.com/products/products/satabeast/ satabeast.html

If I do the math, a 48U rack can hold 12 of these 4U devices. Filled with 500GB drives, that's about 20TB of disk space per device. A rack full of them would be 240TB. That would be $4.17 per TB per month to house the disk space. I think that's pretty reasonable for over a half million dollars in hardware.

The Internet Archive Project http://www.archive.org/ is running on the PetaBox http://petabox.com/ rack system, which was commercialized by Capricorn Tech http://www.capricorn-tech.com/ more than a year ago.

This system uses absolutely no board/controller lever redundancy, instead they use a separate file system on every disk, then mirror pairs of 1U units, and finally mirror the entire (mirrored) rack to a geographically distant location.

I am currently testing a much denser solution, the SATABeast http://nexsan.com/products/products/satabeast/sata beast.html from nexsan http://nexsan.com/ which manages to pack 42 500 GB SATA drives into a single 4U rackmount box. With multiple RAID5 volumes and shared hot spare drives, this results in about 17-18 TB of usable file system space.

According to the nexsan engineer I spoke with today, they do so much burn-in testing of the Hitachi Deskstar drives they ship, that over the 15-18 month period they've used these drives, the total error rate has been just 0.4%.

Even if these numbers are somewhat skewed due to many systems (i.e. drives) being relatively recently installed, it is still very impressive.

For our setup we plan to use multiple full boxes, each connected to a separate NFS server. Each server has multiple FC host adapters, so if a server crashes, the corresponding box can be connected to one of the other servers.

We will also use rsync to mirror all data across the country to a secondary site.

Terje

The Internet Archive Project http://www.archive.org/ is running on the PetaBox http://petabox.com/ rack system, which was commercialized by Capricorn Tech http://www.capricorn-tech.com/ more than a year ago.

This system uses absolutely no board/controller lever redundancy, instead they use a separate file system on every disk, then mirror pairs of 1U units, and finally mirror the entire (mirrored) rack to a geographically distant location.

I am currently testing a much denser solution, the SATABeast http://nexsan.com/products/products/satabeast/sata beast.html from nexsan http://nexsan.com/ which manages to pack 42 500 GB SATA drives into a single 4U rackmount box. With multiple RAID5 volumes and shared hot spare drives, this results in about 17-18 TB of usable file system space.

According to the nexsan engineer I spoke with today, they do so much burn-in testing of the Hitachi Deskstar drives they ship, that over the 15-18 month period they've used these drives, the total error rate has been just 0.4%.

Even if these numbers are somewhat skewed due to many systems (i.e. drives) being relatively recently installed, it is still very impressive.

For our setup we plan to use multiple full boxes, each connected to a separate NFS server. Each server has multiple FC host adapters, so if a server crashes, the corresponding box can be connected to one of the other servers.

We will also use rsync to mirror all data across the country to a secondary site.

Terje

I am currently piloting a setup intended to replace 35 TB of disk+tape used for sesmological data:

Each array will consist of a pair of full (42-disk) nexsan SATABeast 4U boxes. These use RAID5 + hot spares and connect to the host systems with FC.

The two pairs will be mirrored (over a fiber connection) between two geographically separated locations.

The payback time for this setup is so short that we can plan on replacing half the gear every year, using a staggered schedule. (Replace A after year 1, B after two years, then A/B/A/B etc.)

Terje

I've created enormous filesystems (500TB and up) with the following recipie:

As many ATABeasts as you can afford (they'll currently do about 20TB each)...

As many QLogic 2GB fibrechannel cards as you have ATABeasts

1 or more Sun V880s (Just keep buying them as you fill PCI slots)

Veritas Volume Manager/Veritas Filesystem 4.1 or higher.

NFS/Samba to your heart's content

Now, I realize that there are two glaring problems with the above recipie. The first is that only Samba is open source, and I understand theres something of an issue about slashdotters using software that they actually have to pay for. Fear not--in this case the result will be software that actually works.

The second is that Sun will only look at 2TB LUNS and no bigger---and Veritas will only see ~1 TB "drives" or smaller. This actually isn't a problem--simply configure your ATABeasts intelligently, throw a fuckload of LUNS at Solaris (It can take it) and smoosh them all together into one enormo-volume with Veritas.

Nexsan has a box called ATA Beast
Raid, Fibre Channel, 42 ATA drives per 7 RU chasis. Throw in 500GB drives and 1 parity drive for every 6 data drives and you have ~30 TB per chasis.

Just in the interest of science, this posting is a test of this Google-boosting strategy. Now if everybody on /. replies to my post, my company's page ranking may start to approach that of the professional index-spammers.

Seriously, I wonder why Google can't just filter out links to sites which don't have any relevant key words....

If you want to spend the extra money and have a warranty and fancier case, look at Nexsan , or EMC's AX100. Scary that EMC is selling something cheaper than the competition, but they are. Sorta disturbs the natural order of the universe. Still, either will set you back several thousand. The AX100 looks pretty impressive on paper. Options for dual controllers, and up to 3 TB in a 2U space. Haven't tried one myself yet.

Disclaimer: I work for a storage integrator, both are brands we sell.

The Arena isn't as good as the Nexsan. I've seen similar Nexsan units go for $2K on ebay. I highly recommend them. Easily does 70M/sec with RAID5.

My Arena uses 120GB WD's and the Nexan uses 300GB Maxtor's. I have a deployed Arena 8 drive unit with 250GB Maxtor's serving as a live backup for a network. Paid less than $800 for that unit.

http://www.nexsan.com/
http://www.maxtronic.com/

So, you can put your data on 4-5 HD's, 10 tapes or 232 DVD's per month. The Cost of doing so will be about $500 per month for the tapes or HD's and $50 for the DVD's (assuming your time cost $0)

At work, we had a need to keep a few TB of data online permanently, so we purchased a few NexSAN ATABeast's. At $50,000 for 10TB of usable storage ($5/GB), they may be a bit out of your price range. The advantage is that you can hold almost a years worth of data and it is protected by RAID5. It also makes management a lot easier, since it is very difficult to mount 42 300G drives in a single chassis (and it takes only 4U of rack space).

On the low end, NexSAN has the ATABoy2 or ATABaby (2TB or 1TB) for the $8-$15K range. This will let you hold a months worth of data

On the high end, You have EMC disk arrays (Think upwards or $20+/GB for the 'cheap' stuff from them.

Overall, if you have 1TB per month, you need to either a) get a grant to fund your work, b) hire somebody to swap DVD's for you or b) seriously rethink your data generation.

Any of the "cheap" storage methods have serious drawbacks, and the low cost ones are, well, not so low cost if $15,000 sounds like a lot of money to you.

otherwise, good luck

why not just RAID a bunch of smaller cards together?

You laugh, but I am tempted to do this very thing. Since CF cards conform to the IDE interface, theoretically you could put 14 of them onto one of these. Oracle log files, which demand fast access, but not super high transfer speeds, would benefit greatly.

Many companies are selling IDE-based RAID boxes. For instance, take Nexsan's ATABoyII (Sponsored link). It has 1TB of usuable storage (hardware RAID-5 with hot-standby disk and battery-backed cache, redundant power supply and fans). The FCAL version is I believe close to $18,000 each. So 50 TB would be $920,000. Then you need FCAL switches, fiber optic lines, and a few servers to serve the data. Overall, including power conditioning, air conditioning etc, I think $20M is overkill. They're probably going with EMC or Hitachi, which have very nice to configure (GUI), mostly reliable but completely overpriced arrays, and in this case $20M looks right (including consulting fees).