Slashdot Mirror
Does ZFS Obsolete Expensive NAS/SANs?
hoggoth writes "As a common everyman who needs big, fast, reliable storage without a big budget, I have been following a number of emerging technologies and I think they have finally become usable in combination. Specifically, it appears to me that I can put together the little brother of a $50,000 NAS/SAN solution for under $3,000. Storage experts: please tell me why this is or isn't feasible." Read on for the details of this cheap storage solution.
Get a CoolerMaster Stacker enclosure like this one (just the hardware not the software) that can hold up to 12 SATA drives. Install OpenSolaris and create ZFS pools with RAID-Z for redundancy. Export some pools with Samba for use as a NAS. Export some pools with iSCSI for use as a SAN. Run it over Gigabit Ethernet. Fast, secure, reliable, easy to administer, and cheap. Usable from Windows, Mac, and Linux. As a bonus ZFS let's me create daily or hourly snapshots at almost no cost in disk space or time.
Total cost: 1.4 Terabytes: $2,000. 7.7 Terabytes: $4,200 (Just the cost of the enclosure and the drives). That's an order of magnitude less expensive than other solutions.
Add redundant power supplies, NIC cards, SATA cards, etc as your needs require.
Get a CoolerMaster Stacker enclosure like this one (just the hardware not the software) that can hold up to 12 SATA drives. Install OpenSolaris and create ZFS pools with RAID-Z for redundancy. Export some pools with Samba for use as a NAS. Export some pools with iSCSI for use as a SAN. Run it over Gigabit Ethernet. Fast, secure, reliable, easy to administer, and cheap. Usable from Windows, Mac, and Linux. As a bonus ZFS let's me create daily or hourly snapshots at almost no cost in disk space or time.
Total cost: 1.4 Terabytes: $2,000. 7.7 Terabytes: $4,200 (Just the cost of the enclosure and the drives). That's an order of magnitude less expensive than other solutions.
Add redundant power supplies, NIC cards, SATA cards, etc as your needs require.
Also should be noted that FreeBSD has added ZFS support to Current (v7). It's built on top of GEOM too so if you know what that is you can leverage that underneath zfs.
Not enough specifics here. I am going to say do your thing. If it works, you're a hero and saved 47k. If it doesn't obfuscate and negotiate the 50k of storage down to 47k. Win for all.
Unless you would like to give more specifics. Cause I am going to say in 99% of cases where you want fast, reliable, and cheap storage you only get to pick two.
Porn jokes aside, what in the World does a common "everyman" need with that kind of storage?
I have a 40 gig OEM drive on this machine that I've had since 2003, and I still haven't approached the half way mark. And I run a couple of businesses.
I prefer Flambe as apposed flamebait.
For quite a while now, it has been less expensive to build a DIY file server then to purchase NAS equipment. I personally build gateway/NAS products using Via c7/8 boards as they are low power, have hardware encryption, and are easy to work with under linux. Accessory companies even make back plane drive cages for this purpose that fit nicely into commodity PCs.
BBH
place where i work looked at one of these things from another company. did the math and it's too slow even over gigabit for database and exchange server. OK for regular file storage, but not for heavy I/O needs
Does the overuse of TLAs obfuscate the meaning of SDS?
Infiltrated dot Net
For starters, our SAN uses extremely fast connectivity. It sounds like you're moving your disk I/O over the network, which is a fairly significant bottleneck (even Gb). We also have the flexibility of multiple tiers - 1st tier being expensive, fast disks, and 2nd tier being cheaper IDE drives. I imagine you can fake that a variety of ways, but it's built in. Finally, there's the enclosure itself, with redundant power and such.
Still, I bet you could do what you want on the cheap. Being in health care, response time and availability really are life-and-death, but many other industries don't need to spend the extra. Best of luck.
A good 20k$ RAID array does much more. First, it doesn't use cheap SATA drives, but Fiberchannel Drivers or even SAS drives which are tested to a higher level of quality (each disk costs like 500$ or more..). And those cheap SATA drives also react much more poorly to non-sequential access (like when you have multiple users). They are unusable for serious file serving. You can never compare RAID arrays that use SATA/IDE to ones that use enterprise drives like FC/SCSI/etc, because the drives are quite different.
Then you have the other features like dual redundant everything: controllers, power supplies, etc. Then you have thermal capabilities of rack-mount solutions that often are different from SATA, etc, etc.
Do you use consumer level drives, or enterprise level drives? You have not specified that. The cost varies.
http://milek.blogspot.com/2007/04/hw-raid-vs-zfs-s oftware-raid-part-iii.html
Infiltrated dot Net
Well I must say it's true: a company is born on /. every minute.
Religion is what happens when nature strikes and groupthink goes wrong.
Speaking from personal experience - This file system is far from ready. It can kernel panic and reboot after minor IO errors, we were hosed by it, and probably won't ever revisit it. This phenomenon can be repeated with a usb device, you might want to try it before you hype it. Try a google search on it and see what you think...there is no fsck or repair, once it's hosed, it's hosed, the recovery is to go to tape. http://www.google.com/search?hl=en&q=zfs+io+error+ kernel+panic&btnG=Google+Search
ZFS has a lot of potential. However the current implementation of ZFS has its limits, and you should know what they are before you commit to maintaining a server running it.
09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
I'd say this highly depends on your usage of the NAS/SAN. I, for example, have setup a FreeBSD NAS (Samba Share) of a Software Raid-3 configuration. For the most part this is mainly a storage setup (write-once, read often). This coupled with my Gb network works well enough for me. It's fast enough that I'm able to burn dvds over the network and look at files in real time (and watch videos without any lag). If you're looking to implement a high volume of reads/writes using your setup, you should look into using a hardware based raid configuration (Raidz is nice and all but the parity is still calculated in software which can take up more cycles than necessary). I see absolutely no point in buying a pre-made NAS/SAN as all they've done is pre-set everything up.
Again, determine your situation, if it's mainly storage, than your raidz1 setup should be fine. But if you're going for very high I/O I would recommend using a hardware-based raid setup. In either case Gb ethernet should be fast enough (make sure you use a cat 6 and not a cat 5e)
Businesses buy SANs to consolidate storage, placing all their eggs in one basket. They need redundant everything, which this doesn't have. Additionally, SATA drives are not as reliable long term as SCSI. Compare the data sheets for Seagate drives, they don't even mention MTBF on the SATA sheet.
Businesses also want service and support. They want the system to phone home when a drive starts getting errors, so a tech shows up at their door with a new drive before they even notice there are problems. They want to have highly trained tech support available 24/7 and parts available within 4 hours for as long as they own the SAN.
Finally, the performance of this solution almost certainly pales as compared to a real SAN. These are all things that a home grown solution doesn't offer. Saving 47K on a SAN is great, unless it breaks 3 years from now and your company is out of business 3 days waiting for a replacement motherboard off Ebay.
That being said, everything has a cost associated with it. If management is ok with saving actual money in the short term by giving up long term reliability and performance, then go for it. But by all means, get a rep from EMC or HP in so the decision makers completely understand what they're buying.
In a standard corporate environment where the thought of spending $50,000 has come to the plate, you're probably in deeper than you think. A lot of what you pay for with a big name SAN from EMC or the like is that you're getting serious support and reliable equipment from a well established company. Your homebrew method will almost undoubtedly work... but all equipment fails, and when this solution fails, unless you're picky about getting manufacturer warrantees, then you're going to be losing money to fix the problem. With our EMC/Dell solution, if something goes wrong we'll have a tech with a replacement on site within 4 hours. If your thing fails, can you get an RMA replacement in even under a week?
I'm not trying to discourage you from building this system, in fact I think DIY is a great way to go. However, you do need to take into account how downtime will affect the cost of this device. It is always important to have a failover/replacement plan for when your system goes down because most systems DO go down. (Which is why many of us are even employed.)
Good luck to you, sir!
ZFS does not obsolete NAS/SAN. However, for many many many instances, DIY fileservers have been more appropriate than SAN or NAS situations for many concepts long before ZFS came along, and ZFS has done little to change that situation (though adminning ZFS is more straightfoward and in some ways more efficient than the traditional, disparate strategies to achieve the same thing).
I haven't gotten the point of standalone NAS boxes. They always were not fundamentally different from a traditional server, but with a premium price attached. I may not have seen the high-end stuff, howerver.
SAN is an entirely different situation all together. You could have ZFS implemented on top of a SAN-backed block device (though I don't know if ZFS has any provisions to make this desirable). SAN is about solid performance to a number of nodes with extreme availability in mind. Most of the time in a SAN, every hard drive would be a member of a RAID, with each drive having two paths to power and to two RAID controllers in the chassis, each RAID controller having two uplinks to either two hosts or two FC switches, and each host either having two uplinks to the two different controllers or to two FC switches. Obviously, this gets pricey for good reason which may or may not be applicable to your purposes (frequently not), but the point of most SAN situations is no single-point of failure. For simple operation of multiple nodes on a common block device, HA is used to decide which single node owns/mounts any FS at a given time. Other times, a SAN filesystem like GPFS is used to mount the block device concurrenlty among many nodes, for active-active behavior.
For the common case of 'decently' availble storage, a robust server with RAID arrays has for a long time been more appropriate for the majority of uses.
XML is like violence. If it doesn't solve the problem, use more.
It's been a couple of years since we built our first multi-terabyte storate (~1.6TB for ~$5k US) and started the grand experiment with XFS. Of all we've learned, I think the biggest lesson has been that XFS has problems in this realm. We are now spinning over 30TB and likely to double soon. To SAN or not to SAN has become the question, as finding our data and metadata are both important to us, more important than slinging the files off and accidentally discovering them later.
Some form of indexing system is valuable, especially if you're looking at multiple volumes to span.
We're now looking at lustre and possibly zfs to support our solutions. For hardware, we've gone to ATA-over-Ethernet with CoRAID hardware and been pretty satisfied. It's not iSCSI but it works well and we get adequate transfer rates. We do a caching process when we're anticipating high user access periods and can predict the patterns.
I'd say, for the money, try it, benchmark it, and report back.
Never ascribe to malice that which can adequately be explained by tenure.
NAS and SAN are two very different technologies with different goals.
iSCSI is a block level protocol that requires an iSCSI soft initiator or iSCSI HBA ToE for other computers to access it. It is a similar but slow, less secure, and cheaper solution than a normal fibre channel SAN.
To do it right you really should have an iSCSI HBA ToE in the target and initiator, as well dedicated router that is made to handle iSCSI - because of its inherent "bursty" nature allot of routers choke on it.
NAS is file level transfer that uses NFS and CIFS which are already built into every OS
I have experience with a number NAS solutions and if cost wasn't or reliability/throughput was paramount I would continue to purchase them (e.g., Netapp). Depending on the environment they are being installed in the (perceived) liability and additional complexity can be challenging to overcome.
With that said for places where rolling your own is an option I would keep your eye out for a good deal on drives and you will be able to build one much less expensive. I put together a new Myth backend with the following:
Antec Sonata II - $65 (rebate)
Asus M2N32-Vista addition (it's running Liux but the vista addition has an LIRC supported IR receiver) - $210
AMD 4200+ X2 - $96
2GB RAM - $55
Nvidia 7600 with HDMI out - $110
6 x 500GB Maxtor SATA II HDDs - $600
It's not RAID-Z but with a standard RAID-5 I have 2.5TB usable storage with HDTV output and ATA/iSCSI targets for $1136. Not bad and Linux SW RAID-5 write speed actually screams these days, with this setup I expect 200MB write throughput.
One word of caution with RAID-Z, although writes are extremely fast there is a performance issue around reads if they are small and random because there will be a lot of cache misses. Relatively speaking it's not that bad but something to kep in mind when looking at the workload you will be supporting.
It uses SATA drives (assuming it's big enough to be called an array, rather than being five disks shoved in a 1U box). If you want FC or SAS, you're looking at $50K on up -- probably up.
There is nothing new here; it has always been cheaper to work out your own storage solution than to buy a commercial unit. Same goes for servers/desktops etc. Why buy a Sun (or Dell, or IBM or whatever) when you can get your local guy to put a system together for half the price?
If you are an individual or even a small business with limited capital then DIY is often the best (only?) way to go but you also get to deal with flakey controllers, incompatible drivers, and warranty returns all on your own. The integration of components, performance management, and the harmony of the complete system is all yours.
At some point, either because of the scale or the criticality of the system, it is worth the bucks to pay someone who has researched the issues and built a solid product to provide you with a solution that you can (hopefully) trust. Your sysadmins and techies can spend their time on ROI generating projects instead of figuring out why a component does wild and whacky crap every full moon. Tech support can be a very good thing.
Even open source has its commercial providers. Personally, I have always liked Slackware but if we are deploying servers it's going to be Red Hat.
I think homebrew is super: put your system together and do some benchmarking, then publish it for the rest of us to benefit!
It's no NetApp - yet. One thing to realize is that iSCSI target isn't even in Solaris proper yet - you have to run Solaris Express or OpenSolaris for the functionality. That may be fine for some people, but it's a deal-breaker for most companies - you're really going to place all those TB of data on a system that's basically unsupported? I'm sure Sun would lend you a hand for enough money, but running essentially a pre-release version of Solaris is a non-starter where real business is concerned. Even when iSCSI target makes it into Solaris 10 - which should be in the next release - are you really comfortable running critical services off of essentially the first release of the technology? Furthermore, while ZFS is amazingly simple to manage in comparison to any other UNIX filesystem/volume manager, it still requires you to know how to properly administer a Solaris box in order to use it. Even GUI-centric sysadmins are generally able to muddle through the interface on a Filer, but ZFS comes with a full-fledged OS that requires proper maintenance. Your Windows admins may be fine with a NetApp - especially with all that marvelous support you get from them - but ask them to maintain a Solaris box and you're asking for trouble. Not to mention, since it's a real, general purpose server OS, you'll have to maintain patches just like you do on the rest of your servers - and the supported method for patching Solaris is *still* to drop to single user mode and reboot afterwards (yes, I know that's not necessarily *required*). Also, "zfs send" is no real replacement for snapmirrors. And while ZFS snapshots are functionally equivalent to NetApp snapshots, there is no method for automatic creation and management of them - it's up to the admin to create any snapshotting scheme you want to implement. Don't get me wrong - I love ZFS and I use it wherever it makes sense to do so. It may even be acceptable as a "poor man's Filer" right now, assuming you don't need iSCSI or any of the more advanced features of a NetApp. In fact, it's a really great solution for home or small office fileservers, where you just need a bunch of network storage on the cheap - assuming, of course, that you already have a Solaris sysadmin at your home or small office. Just don't fool yourself, Filer it ain't - at least not yet.
I have a terabyte RAID in my main computer, and a 2 TB fileserver. With my video editing, I'm starting to look at a full fileserver (still have most of my main box empty). And, I'm not even a pirate, and I've just been a computer and video hobbyist for 2 decades now. I'm not even that serious about my hobbies.
I can imagine we'll all need terabytes of capacity in the next few years. Some games are already into the 14-20GB sizes.
based on progression of drive sizes, A 1.3+ terabyte drive should be available within a few years, and a 7.0+TB drive shortly thereafter. If you want cheap storage, just use large single drives. They are super cheap and while not super large, they are far less complicated than this setup.
stuff |
Whether you buy or build depends a lot on whom you're buying from. Buying from people who are not in the storage business, even if it is a big corporation like Dell, gives you about the same level of support as rolling your own when the shit hits the fan. Don't believe me? See this one, and notice how long the problem kept on and on (me is one of the happy users there):
m essage?board.id=pv_raid&message.id=214&view=by_dat e_ascending&page=1
http://www.dellcommunity.com/supportforums/board/
Buying from EMC or the likes (even EMC from Dell) tends to work better. The hardware is expensive, the consulting fee is expensive, and the support is expensive, but at least for that kind of money you are sure someone tries a bit harder to help.
All in all, it depends on your business. If you are making a zillion a month from that hardware working flawlessly, _not_ paying $200k for the storage is dumb. If you are making little enough so that $50k makes you think about it, rolling your own could be the way to go.
This doesn't strike me as having much to do with ZFS at all. You've been able to do a home grown NAS / SAN box for years on the cheap using commodity equipment. Take ZFS out of the picture and you just need to use a hardware raid controller or a block level RAID (like dmraid on Linux or geom on FreeBSD). There are even canned solutions for this, like OpenFiler.
That being said, this sort of solution may or may not be appropriate, depending on site needs. Sometimes support is worth it.
You're also grossly overestimating the cost of an entry-level iSCSI SAN solution. Even going with EMC, hardly the cheapest of vendors, you can pick up a 6TB solution for about $15k, not $50k. Go with a second tier vendor and you can cut that number in half.
Ah yes, digital photography. It's a good thing I asked because I'm in the (gradual) process of moving away from film. Which means, I'll be having a similar problem as yourself. If you do it, I hope you post your results. I will be really interested!
BTW, a 5 MB photo file sounds very small - even if it's jpg. I'm assuming you made a typo and your storing a 2 - 5 megapixel image, which would be, what, at least 15 megabytes? Even more reason for the setup you're talking about!
I prefer Flambe as apposed flamebait.
How is it with redundancy? you got redundant PSU, redundant controlers? redundant Network-paths to and from the server. Obviously the tech itself have some LARGE advantages. But working with enterprise technology makes you think redundancy*3. (No one wants the SQL-cluster to fail due to a failed PSU that turned out to be a single point of failure.)
"Never EVER mess with a jumper you don't know about, even if it's labeled 'sex and free beer'." - Dave Haynie
Some of these issues looked familiar, so I thought I'd do a basic comparison:
Reiser4 had the same problems with fsync -- basically, fsync called sync. This was because their sync is actually a damned good idea -- wait till you have to (memory pressure, sync call, whatever), then shove the entire tree that you're about to write as far left as it can go before writing. This meant awesome small-file performance -- as long as you have enough RAM, it's like working off a ramdisk, and when you flush, it packs them just about as tightly as you can with a filesystem. It also meant far less fragmentation -- allocate-on-flush, like XFS, but given a gig or two of RAM, a flush wasn't often.
The downside: Packing files that tightly is going to fragment more in the long run. This is why it's common practice for defragmenters to insert "air holes". Also, the complexity of the sync process is probably why fsync sucked so much. (I wouldn't mind so much if it was smarter -- maybe sync a single file, but add any small files to make sure you fill up a block -- but syncing EVERYTHING was a mistake, or just plain lazy.) Worse, it causes reliability problems -- unless you sync (or fsync), you have no idea if your data will be written now, or two hours from now, or never (given enough RAM).
(ZFS probably isn't as bad, given it's probably much easier to slice your storage up into smaller filesystems, one per task. But it's a valid gotcha -- without knowing that, I'd have just thrown most things into the same huge filesystem.)
There's another problem with reliability: Basically, every fast journalling filesystem nowadays is going to do out-of-order write operations. Entirely too many hacks depend on ordered writes (ext3 default, I think) for reliability, because they use a simple scheme for file updating: Write to a new temporary file, then rename it on top of the old file. The problem is, with out-of-order writes, it could do the rename before writing the data, giving you a corrupt temporary file in place of the "real" one, and no way to go back, even if the rename is atomic. The only way to get around this with traditional UNIX semantics is to stick to ordered writes, or do an fsync before each rename, killing performance.
I think the POSIX filesystem API is too simplistic and low-level to do this properly. On ordered filesystems, tempfile-then-rename does the Right Thing -- either everything gets written to disk properly, or not enough to hurt anything. Renames are generally atomic on journalled filesystems, so either you have the new file there after a crash, or you simply delete the tempfile. And there's no need to sync, especially if you're doing hundreds or thousands of these at once, as part of some larger operation. Often, it's not like this is crucial data that you need to be flushed out to disk RIGHT NOW, you just need to make sure that when it does get flushed, it's in the right order. You can do a sync call after the last of them is done.
Problem is, there are tons of other write operations for which it makes a lot of sense to reorder things. In fact, some disks do that on a hardware level, intentionally -- nvidia calls it "native command queuing". Using "ordered mode" is just another hack, and its drawback is slowing down absolutely every operation just so the critical ones will work. But so many are critical, when you think about it -- doesn't vim use the same trick?
What's needed is a transaction API -- yet another good idea that was planned for someday, maybe, in Reiser4. After all, single filesystem-metadata-level operations are generally guaranteed atomic, so I would guess most filesystems are able to handle complex transactions -- we just need a way for the program to specify it.
The fragmentation issue I see as a simple tradeoff: Packing stuff tightly saves you space and gives you performance, but increases fragmentation. Running a defragger (or "repacker") every once in awhile would have been nice. Problem is, they never got one written. Common UNIX (and Mac) philosoph
Don't thank God, thank a doctor!
You have a good point about the support with higher value equipment, but at this price he can afford to keep a few spares in the closet, or even have a few other complete units as a failover 'live' backup.
I guess this setup could replace some people's need for a turnkey NAS solution. But your thinking it could replace SAN solutions shows you haven't looked into SAN too much. To start, there's a reason Fibre Channel is way more popular than iSCSI. The financial services company I work for has about 3 petabytes of SAN storage, and not a drop of it is iSCSI. Storage Area Networks are special built for a purpose. They typically have multiple fabrics for redundancy, special purpose hardware (we use Cisco Andiamo, i.e., the 9500 series), and a special purpose layer 2 protocol (Fibre Channel). iSCSI adds the overhead of TCP/IP. TCP does a really nice job of making sure you don't drop packets, i.e. layer 3 chunks of data, but at the expense of possibly dropping frames, i.e. layer 2 data. The nature of TCP just does this, as it basically ramps up data sending until it breaks, then slows down, rinse and repeat. This also has the effect of increasing latency. Sometimes this is okay, people use FCIP (Fibre Channel over IP), for example. But, sometimes it's not. Fibre Channel does not drop frames. In addition, Fibre Channel supports cool things like SRDF which can provide atomic writes in two physically separate arrays. (We have arrays 100 km away from each other that get written basically simultaneously and the host doesn't think its write is good until both arrays have written it.) So, like I said, this might be good for some uses, but not for any sort of significant SAN deployment.
"Nature doesn't care how smart you are. You can still be wrong." - Richard Feynman
It's a pity your summary fails to emphasise: Most of these are known bugs; some are likely already fixed; and a couple are simply wrong.
But it's all beside the point. ZFS still offers data integrity that no other hardware or software system does (some other systems do provide copy on write, pool-like manageability, and so on). One day we'll all be using it (or a clone).
you had me at #!
Google have a great solution that focuses on the “cheap” part without compromising much the latter two. If you have not read up on the Google Filesystem, definitely take the time to. At the very least, it seems to call into question the need to shell out tens of thousands for high-end storage solutions that promise reliability in proportion to the dollar.
Why bother.
I would go with the Dell EMC AX150i SP Array for an iSCSI solution of that size - it can do iSCSI up to 6TB.
http://www.leadmagnet.50megs.com
Potentially it will obselete low-end NAS/SAN hardware (eg: Dell/EMC AX150i, StoreVault S500) in the next couple of years, for companies who are prepared to expend the additional people time in rolling their own and managing it (a not insignificant cost - easily making up $thousands or more a year). There's a lot of value in being able to take an array out of a box, plug it in, go to a web interface and click a few buttons, then forget it exists.
However, your DIY project isn't going to come close to the performance, reliability and scalability of even an off the shelf mid-range SAN/NAS device using FC drives, multiple redundant controllers and power supplies - even if the front end is iSCSI.
Not to mention the manageability and support aspects. When you're in a position to drop $50k on a storage solution, you're in a position to be losing major money when something breaks, which is where that 24x7x2hr support contract comes into play, and hunting around on forums or running down to the corner store for some hardware components just isn't an option.
ZFS also still has some reliability aspects to work out - eg: hot spares. Plus there isn't a non-OpenSolaris release that offers iSCSI target support yet AFAIK.
I've looked into this sort of thing myself, for both home and work - and while it's quite sufficient for my needs at home, IMHO it needs 1 - 2 years to mature before it's going to be a serious alternative in the low-end NAS space.
Buy one of CORAID's 1521 disk shelves w/ their CLN20 front end for $6600 and drop in 15 500 GB SATA drives (they're a whopping $100 each these days) for a quick 7TB of raw storage for ~$8K or ~9K. Need more storage? Go w/ 750GB drives (They're validating the 1 TB raw drives now, but the price isn't worth it, per GB). Want to add storage later? Buy another 1521 and plug it in. Oh, and it's AoE, with less overhead than iSCSI.
Linux has more perfomance testing on x86 than OpenSolaris (so you are not as likely to run into a bad bottleneck). On Linux you can create a RAID-1,-4,-5 and -6 under Multiple Device Driver Support in the kernel. You can then use mkraid to include all the drives you want. This code in not new at all. It was stable in 2.4, maybe even in 2.2
After that you just create a filesystem on top of the raid. If you don't like ext3 or don't trust it, there is always xfs. I had some rough times with reiserfs, xfs, and ext3 and for all the experience I had I would go xfs for long running server environments (and now get flamed for this little bit, use ext3 all you want).
The advantage is that you use very well tested code.
The problem comes with hotswapping. I don't know if the drivers are up to that yet. But I also highly doubt that OpenSolaris SATA drivers for some low price chip in a low price storage box can deal with hotswapping. So Linux might be faster on that one.
That is a setup I would compare to a plug'n play SAN solution. And it totally depends on the environment. If the Linux box goes down for some reason for a couple hours/days, how much will that cost you? If it is more than twice the SAN-solution, you might just buy the SAN and if it fails just pull the disks and put them in the new one. I dunno if that would work on Linux.
Ever heard of Starfish? It's a new distributed clustered file system:
Starfish Distributed Filesystem
From the website:
And you can build clusters at relatively low cost:
(warning: I work for the company that created Starfish)
-- manuManu Sporny (skype: msporny, twitter: manusporny, G+: +Manu Sporny)
Founder/CEO - Digital Bazaar, Inc.
Depends on what you mean by "generic" components. There are plenty of generic servers cases out there designed for storage applications. A 2U rack enclosure with 12 drive bays can be had for around a grand. Add a twelve port raid controller for six or seven hundred and you've got capacity to expand to 6TB raw even if you stick to 500GB drives (currently best price per GB).
I was reading the various comments and I was surprised that nobody suggested a hardware raid ctrl. There is some from well known business (Adaptec, promise and so on) that will support from 4, 8 and more SATA drives and do RAID 1/5/1+0. They usually support hot spare, raid level change, volume growth, etc. These are available from about 300$ with a median for 8 drives at about 500$. Using this kind of solution would be faster and more robust (make sure you are taking a full RAID chipset, not just a RAID accelerator where a combination of software and hardware is necessary) and should be easier to manage as well then a zfs setup over OpenSolaris. These are also often available for multiple OS. I'm looking at this kind of solution for my personal LAN. I didn't had time yet to order the hardware and the disks so I have no experience doing it but on paper, it look good. At this point I'm looking for an Adaptec 2820SA which support 8 drives and offer a lot of interesting features. Does anybody have comments on taking that road or on the Adaptec 2x20SA particularly?
Comment removed based on user account deletion
Once upon a time a new outside broadcast truck was being built. The new boss made changes to the design to use domestic televisions instead of $5000 D1 monitors. Guess what? The teevee sets fell apart and the boss became laughing stock. The proper monitors had to be put in, at great expense.
Computer gear is a bit different. Six years ago a 100Gb real-time disk array cost $100000. A cheaper 100Gb of storage could then have been built with a Pentium PC and four 33Gb IDE Drives, raided together to set you back $3000 or so. A spare could have been built for another $3000 and the systems rsync-ed on a daily basis. Perhaps new disks could have been bought in the years between, maybe another $3000 worth. Assuming the 'linux' box did the job and delivered the 1's and 0's quick enough, what would you prefer, $90000 in the bank or a very expensive and a not so powerful real-time disk?
As it worked out we paid for the $100000 box. It did mess up a few times, even needing new drives. It also sounded like a rocket, not indicative of being energy efficient. Not one client gave a damn about what we were using for storage, however, we did have to get them to pay lots of money for such toys and we did have to manage the financial risk.
Looking back I think the homebrew solution could have been more fun, perhaps to give a business edge and an opportunity to gain useful skills instead of niche knowledge.
Clearly the solution has to suit the application, and in this case some budget could be spent on cache RAM, hopefully to provide a really interesting setup. If things are DB intensive then some of the indexes could stay in the cache-RAM, maybe to give better performance than with the $50000 SAN and with no application re-writes.
I have one Linux fileserver with 5TBs for some time now, the only issue ever was a dead PSU. Fixed that by replacing Fortron with Enermax.
Only thing I would do differently now is to use PCI-E SATA controllers for bandwith (my server has PCI). Linux software RAID is perfectly up to the task. ZFS should also be.
One thing: Do temperature and SMART moniroting on all drives and run along SMART selftest every two weeks sor so. Also have the RAID and SMART monitoring software send Email on problems and have at least one spare disk ready to use.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
You can't compare zfs to SAN. I didn't need a san but look at nas. Netapp is great but still had a few limitations.
- The discs were proprietary.
- redundancy. To go with the redundant head was to bump up to a new level in cost.
- chassis size was set
- the cost ran up fast for the software to do the things I wanted. all proprietary (which was alright considering)
Still, netapp is quite a quality product.I wanted a cheaper solution that had more to it so I looked at coraid.
- They use an AoE (ata over ethernet) protocol that is lighter than tcp/ip.
- It can run with off-the-shelf sata drives
- a gateway can be used if you want to add more shelves
- they have redundant gateway solution as well
We'll be putting it in this quarter.Having said this, I'm unsure of the state of lvm2 for block level snapshots. This was one thing that netapp did well. Also, be selective on the drives you choose. This is only another option you can look into.
Take a look at this solution: www.openfiler.com. Supports CIFS, NFS, FTP, iSCSI, and WebDAV protocols. Provides snapshots, quotas, and active directory integration. It can use builtin storage or act as a gateway to existing fibre channel or iSCSI storage. Plus, you can receive commercial support for it.
If you need more than 1-3 TB, you can't use generic components
;-)
Why not?
Sure, a 16-channel SATA controller with RAID 0/1/5 will cost you $400. But that will handle, using 750GB drives that have recently entered the "affordable" range, a total of 12TB (or more practically, a 10.5TB RAID5 with one hot spare). Find an OEM that can set you up with that for under $5000 total.
Now, that uses a PC chassis and wouldn't look "nice" in a rack. So what? If you need 10TB and don't want to blow $50k on it, you don't have a lot of choices... So if you insist on all racked equipment, buy a rack shelf kit and lay it on its side (and hide it with blanks if you care that much)
Don't forget Zumastor (http://code.google.com/p/zumastor/).
i cation.html )
It's kind of like a homegrown subset of ZFS, but
it already has online remote replication, which
is one of the key features of Netapp NAS boxes.
(People are starting to add that to zfs; see also
http://milek.blogspot.com/2007/03/zfs-online-repl
Does Vista run ZFS yet? I know Apple isnt supporting ZFS until 10.5, I have a NAS that uses the ZFS file system and I have had limited use of it because I cannot access it on half of my hardware due to the os.
I've been using ZFS for quite some time, and have yet to have any form of failure or data corruption. I've used it with simple JBOD, SAN attached, even USB attached drives - it just plain works. With Solaris 10 U3, and the latest revs of OpenSolaris, you have access to RAIDZ2 - which gives you double parity, even more protection. Snapshotting can be scripted to run as often as you like. I keep 2 months worth of snapshots every 5 minutes day in and day out. You can disassemble the system, scramble the drives, reattach and bring the system back up and all will be well. ZFS will just re-assemble the pool and continue. You can replace drives on the fly. Let's say you make your 12 disk encloser with 750GB drives. Now, 2 years later you want to replace them with 2TB drives. Simply use the zfs replace command to replace each drive, wait for it to re-silver and rebuild the data on the replaced drive, then move on to the next drive. As you replace them, the pool will grow automatically. This would grow your (assuming 12x750GB in a RAIDZ2) 7.5TB pool to a 20TB pool, without downtime. With OpenSolaris on x86, you can even boot off of ZFS now, so use ZFS to mirror your boot disk with a like drive and you should be good for quite some time... Using something like Sun's Thumper system, you can get a 12TB system for less than 30k (for those who have something akin to a budget) ZFS, it's fast, safe, secure - and I enjoy working with it (as I don't have to do much with it).
Who is general failure, and why is he reading my hard drive?
Depends on needed storage space. If you need more than 1-3 TB, you can't use generic components, price goes up and hardware starts taking more space than dedicated NAS box. Or Tyan 2U-4U boxes are DIY in your country.
m obilerack/CSE-M35T-1.cfm
With the Via config and the Supermicro drive cages I outlined in my post, you can effectively have 15 drives in a medium tower case. The case and basic components will run you around $250 - 300 US. All that remains is your distro of choice and the HDDs. Grab FreeNAS and 15 1TB HGST drives and you now have a 14TB fileserver.
My main complaint about the Via reference boards is that few of them come with GB ethernet. I usually add a GB ethernet to the single PCI slot. I'd go with an AMD or Intel, but the lack of features and price tend to put me off. The use of SATA port multipliers aleviates the need for HW Raid.
BBH
References
http://www.cooldrives.com/cosapomubrso.html
http://www.supermicro.com/products/accessories/
Ironically, just a few weeks ago our Google Search Appliance lost two drives in a few days (resulting in a new server being shipped to us after some hassle with customer service). One theory is that the drives don't handle losing power very well (we had a power outage). My guess is that under ideal conditions, the failure rates are about the same. When things go wrong (e.g. hotter than normal, power issues, etc), perhaps the SAS drives have an advantage.
On sheer size, yeah, you can cram a lot of drives into it and SAN them together, but I think you're going to find the performance of a low-end iSCSI SAN to be lacking. I use one as the disk storage component of a disk-to-disk-to-tape backup system, and even then it can be a performance bottleneck. It was a $3000 array loaded with $1200 worth of drives, so I'm not complaining, but if you're looking for blazing fast storage, DAS USCSI360 or SAS is gonna whoop its ass.
In other words: You get what you pay for. Capacity, reliability, speed. Pick any two or be prepared to pay 10 times as much.
could be on the way to make NetApp solutions irrelevant in the future. Why pay $$$ when you can get something that does the same thing and much more and that is free? ZFS is showing a lot of promise.
Why? Because it's 128 bits! One hundred and twenty eight fucking bits! That's 64 bits more than any other FS, so any fool can see that it's twice as good as the alternatives.
It may be new and untested, but that's hardly important in the face of 128 fucking bits is it? Besides it's designed by Sun engineers and nobody has more experience in FS design and implementation. That's why all the previous Solaris filesystems rocked so hard. Nothing can beat UFS in terms of stability and performance after all.
Oh, and I nearly forgot - because it's made by Sun it's going to be 10 times as Enterprisey as any half-baked, so-called "tried and tested" RAID/SAN solution that those other suppliers are going to come up with.
Quite frankly, the fact that you're even asking this question suggests you are guilty of criminal hype evasion.
I know there are lots of downsides (thinking loss of the digital advantages) but you make it sound like when it comes to image storage old 35mm negatives seem to be competitive.
After re-reading the article and your comment, I was reminded of this unfortunate fact. Speculating on the reasons, I am not so convinced it is a matter of competitive advantage (as suggested by StorageMojo). Even if someone else set up shop with massive storage using GFS, they still would not offer the services Google provides. And with the exception of Gmail (whose limitless storage is not the most compelling feature), end-users (typically?) do not consider how Google stores data. If I am right, it may only be a matter of time before Google makes GFS available to the masses.
Why bother.
What amazes me is all the talk of iSCSI, but almost no mention of AoE (ATA over Ethernet).
/dev/office. Then I create several lv's (logical volumes) of arbitrary size beneath *that*. So I have /dev/office/home, /dev/office/mp3, /dev/office/blah, etc.
What you have is a box that exports block devices out over layer 2. Another devices loads it as a block device, and can now treat it in whatever fashion it could deal with any other block device, so for example I have 2 "shelves" of Serial ATA drives going. I have a third box that I could either load linux on, using md to create raid sets, or what I've actually done is used the hardware on each of the two shelves, created a raid5 set on each, then used md to create a raid1 set out of the two raid5's. I then take my spankin' new md0 device which is huge for my needs (7.5TB), use LVM to create a volume group (called 'office' for me) and that creates
Now you can format those lv's like any other partition/slice. I've used xfs on all of mine, but you could use ext2/3 if you really wanted.
Karma: Chameleon (mostly due to the fact that you come and go).
Is "Psychiatric help - 5 cents" one of them?
NAS stuff tends to be plug and play. No admins required. Or at least, minimal admins.
Deleted
I remember working with AIX many years ago (that was the time when IBM started exploring Unix and the RISC architecture) and remember very well how with SMIT (their management system) you join all the hard drive into the common pool, and then define the partitions out of that pool. Isn't that something similar to ZFS? If so, then IBM was the first, at least in terms of concept.
...unless your actual net average i/o load profile is fairly light. iSCSI over gigabit ethernet fell on its face under load for us. Trying to mount an MS SQL Server database that lives on an iSCSI target over the network via may for alright for a trivial small number of users hitting a trivially small database with a trivially small I/O load (e.g. less than 25 users hitting an 8GB database with 5-10 queries per minute per user and a fair amount of full table scans due to a stupidly written app that's beyond my control) before the SQL server is brought to its knees waiting for iSCSI I/O over the network. That same database when run on a locally attached Ultra-160 SCSI drive (yesteryear's technology) absolutely smokes the iSCSI setup we tested.
After finding this dismal performance, we knew we could not even think of mounting a much larger Oracle database via iSCSI over gigabit ethernet. That would be a stupid joke.
Simple SMB filesharing (user's Word & Excel docs, PDF's, etc) of our users' network home folders over iSCSI over gig ethernet does work acceptably well, however, since that doesn't really generate all that much network traffic.
In conclusion, I've come to the determination that iSCSI is basically an academic curiosity that was created just because somebody thought it was cool to encapsulate the SCSI protocol in IP packets, and with only a light traffic load, such as a home network or a (very) small business network, and over only gigabit ethernet, the performance is adequate, but why not just use simple and cheap attached big hard drives to your server instead? That is much less complicated.
Maybe with 10 gig ethernet, the performance bottleneck might be less of an impact, but I have no 10 gig hardware to play with.
Your ZFS system could work for SOHO where things such as uptime, disaster recovery, flexible provisioning of disks, speed and support, for instance, are not as critical. Let's look at the list:
1. uptime - so you've raided the box. What happens when the power supply fries or the CPU fails? Redundancy is typically dealt with through multi-pathing in SAN, clustering in NAS.
2. disaster recovery - your box is lost in a flood/earthquake/burglary. Did you have an efficient data remoting solution to a DR site? Putting all this data on DVDs is not practical.
3. flexible provisioning of disks - both SAN and NAS offerings have loads of features to allow provisioning and re-provisioning of storage for different usage. In the NAS world it even goes as far as integration with specific Windows apps (Exchange, SQL, etc.)
4. Speed - Think about a box that can push 1GBytes/sec of data over multiple links from a single filesystem.
5. Support - who's gonna fix the box when it fails or has bugs? Most businesses don't care to do that themselves.
There are many other additional reasons why ZFS+COTS ain't there yet. Where ZFS+COTS is potent is for upstart NAS vendors looking for an easy entry point where a lot of the heavy lifting has already been done. It can be used as a base unto which many features/functionality still need to be added.
In any case, at the low end, you can buy something from Buffalo, Infrant/Netgear or even StoreVault/Netapp if you want a "high-end low-end" box.
I'd like the following scenarios explained.
RAID0 = bunch of hard drives strung together, look like one big drive. in the implementation I'm refering two the data is striped to a block size and written across each disk simeultaneously (or nearly). This is the fastest disk subsystem available but the most susceptible to failure. If one disk fails, you're toast.
Does ZFS do anything in this situation? I have 'one big drive' presented to the operating system, the striping is abstracted at the hardware layer, and I have a semi-expensive ($300 - 800) RAID card running this. In a random I/O workload I get about 150 iops per drive, streaming is another ball of wax typically more interface limited/block size limited than head movement.
To save money, I'd drop the RAID card and put ZFS down. I now have 12 drives (SAS attached), can I get better performance with ZFS like I could with the RAID Card? Think Log Drives for a big DB, or scratch storage space while manipulating a metric assload of video files. Gbit / sec transfer rates for real-time storage of HD Video. In a random I/O workload I get about 150 iops per drive, streaming is another ball of wax.
RAID 0+1. All the perf benefits of RAID0, but 2x the drives. Typically two cabinets RAID 0'd then RAID 1 the two RAID 0s. I get redundancy at a slight penalty of performance due to 2x the writes happening, but no degredation the read.
What can ZFS do for me here? Again, performance improvements/changes?
RAID5
50% penalty in performance even with the best card because in a high drive count you have to read in data, calc, then do the write. However one drive gives me full redundancy, I loose a second though and I'm toast. RAID6 sometimes is used to describe distributing the hot spare into the array, so no more disk space but can take two simultaneous drive failures and keep running.
What does ZFS do for us here?
This isn't a troll, I really know nothing about ZFS and I'm really curious how I could not have to do the above to protect my photography / video data for my photography business. Would be cool if I could do it on my Mac Pro
As a rock-in-roll Physicist once said, No matter where you go, there you are.
How do you back this lot up? :-)
If it weren't for the rocks in its bed, the stream would have no songs.
I just recently acquired a sunfire 280R...and am looking for storage on it. I got a good deal, and have always wanted a Solaris box to play/learn from.
But, from reading about it...it mentions wanting to use FC (Fiber Channel) harddrives...I'm familiar with SATA and IDE...but, the FC ones are new to me..
I've been looking to get a T3 or Storedge array on eBay to hook to it...but, if there were a way to hook a new enclosure like you described, I'd like to go that way. Do you know much about hooking HD's to something like a 280R, or could you point me to links where I can learn more about this?
I got the machine for a steal, but, found it only has 2 HD's in it....and I want to use it as a home server, and want to really add more storage space to it....
Thanks in advance!
Light travels faster than sound. This is why some people appear bright until you hear them speak.........
...everyone stops implementing NAS/SAN and instead implements ZFS. Duh!
There exists no way of exchanging information without making judgments. --Bene Gesserit Axiom
It seems to me that even if the entire setup is prone to failure, all you really need is a gigabit crossover or two running to an identical setup. I don't know if ZFS does anything like this, but I can think of at least one way to make it work on Linux: DRBD + OCFS2 + heartbeat. If you're smart, you can even do some load balancing, at least until one of them fails -- and when that happens, the other should be able to take over very quickly, if not instantly -- Linux heartbeat means it would simply takeover the other machine's IP and start its services.
So, that's $6k total instead of $3k.
The one problem I have with OCFS2 is that when it fences a system, it tends to either bring the whole thing down (kernel panic), or in newer versions, give you the option of forceably rebooting instead. This killed it for a project I was working on, where one of the machines had other mission-critical systems running that were not on the OCFS2, and thus, it seemed retarded to panic and bring down everything else too.
So if that's your problem, you can always build a third, identical system to run the other stuff on. $9k.
Even if you figure another $1k for random stuff, like maybe a LOT of gigabit crossovers, or 10gig fiber, or something, that's still a fifth of the cost of the "business-grade" or whatever else he was considering. Even assuming the worst-case scenario, where the homebrew system costs a lot more to maintain (even electricity and cooling, maybe), how long will it take for it to cost another $40k? And this way, you have an ENTIRELY redundant system -- the only way you lose it is if, say, the whole building blows up.
I mean, I sort of agree that you get what you pay for. But when the difference in price is that much, the only way it's ever worth it is if there's really great support with the high-end package. And is it $40k worth of support? If not, I imagine this guy could put together a company selling little $3k, $6k, and $10k systems for $20k each (including support), shaving off $30k even for the most paranoid.
And all of that is pretending you're right about the cheap consumer-grade hardware actually being less reliable.
Don't thank God, thank a doctor!
i've just recently started looking at centralizing my data at home. i've set up NFS to share a 3ware RAID array over gigabit ethernet, but every time i try to copy some serious volume to the server, i end up with some corrupted files on the receiving end and a dead NFS connection.
currently the server is some older possibly substandard hardware - that could be it - but i'm asking the more general question about NFS.
what does it take to get bullet-proof file sharing for linux clients? is opensolaris the answer? is it more reliable?
free software, open standards, open file formats, no software patents.
I have been trying to convert an old PC to a fileserver, and I haven't yet found a good RAID solution. Some motherboards do provide RAID, but that's software RAID with hardware support. I need a pure RAID 5 solution, does anyone have anything to propose (motherboard/PCI/whatever?). It would help me very much.
Send email from the afterlife! Write your e-will at Dead Man's Switch.
Or, in other words, hell yes, large PC clusters can obsolete mainframes. And yes, it depends how you use it -- or, in other words, if your application actually can support a large PC cluster instead of a mainframe.
Don't thank God, thank a doctor!
Maybe something interesting will happen here: http://code.google.com/p/netgfs/
Why bother.
...between doing this or running Linux instead and using:
1. Either hardware or software RAID for redundancy and performance
2. Using LVM2 to slice up the disks any way that is desired
3. Using NBD or GNDB (network block device or global network block device) to export the LVMs to other systems as block devices or just using Linux's iSCSI implementation or even (yuck) ATAoE
4. Using Samba and/or NFS to share volumes for temporary mappings (as opposed to the permanent mappings with (g)nbd
This is what I was planning on doing. What makes ZFS and RAID-Z better than the above options, which I find to be very easy to implement?
-"...bad old ideas look confusingly fresh when they are packaged as technology" - Jaron Lanier (Digital Maoism on Edge.o
Take a look at http://www.promise.com/product/product_detail_eng. asp?product_id=149
Pretty much what you want...from a decent manufacturer for just $8K (with 15 decent SATA II 250GB drives)
How is this different from Ethernet interface bonding and IP multipathing on Solaris? You can redundant network paths on IP as much as on FC.
With ZFS you also get the added benefit of checksumming (a Merkle tree). No NAS/SAN vender or other file system gives you that.
At home I was running Linux w/ 3 120 GB SATA drives in software RAID-5 and LVM on top. I switched to Solaris 10u3 x86 with ZFS RAIDZ on 4 500GB SATA disks. Much easier to admin the files/resizing plus I get compression and error correction.
My main issues are:
1) An inexpensive ($100) 4 port SATA controller that is supported by Solaris 10u3
I had a $20 card working in Linux. I needed to flash the bios to JBOD (no "RAID" in its bios). I bricked it. I ended up getting a motherboard w/ 4 ports that someone else had already been using/testing. When I add more disk, I'll need to find a controller board. Hopefully 10u4 will be out with support for more boards.
2) Getting lots of disks in an enclosure with power and cooling
The Solaris box got a new case (Antec p180) that has power/cooling for 4 drives in the case. How do I add 4 more? For the other box I built something out of plywood/cardboard to hold the drives and duct the air over them. I added an old PC power supply and a 120 mm fan. 42" SATA cables can run out the back of the PC to it with no problem. I'd like to find something premade that holds 4 + drives and closer to $100.
Any suggestions?
I am wondering what SDS means too. This is the wikipedia entry: http://en.wikipedia.org/wiki/SDS
The minimal FreeBSD distribution, Web interface, PHP scripts and documentation are based on M0n0wall.
http://www.freenas.org
From the ZFS FAQ:
Q: Suppose I have three E450s. Would ZFS allow me to integrate storage across all three boxes into one big "poor man's SAN"?
A: No, ZFS is a local filesystem (for the time being). To access storage attached to a different host, use NFS.
Like administrator skills - the very fact that your asking slashdot about this stuff means that you're inquisitive about this and that puts you far ahead of most of the dum dums I see in the skill shortage world which is IT in Australia at the moment. The number one cause of data loss is a badly skilled administrator. If you understand the system you've deployed and looked through the different scenarios of data loss and have recovery strategies in place, then you've probably covered most of your bases already. And that's the kicker isn't it; it's not so much about what you purchase as it is about having a well thought out process in place when you make one of these decisions. When I was a system admin, the best book I ever read was "the practice of system and network administration" http://www.amazon.com/Practice-System-Network-Admi nistration/dp/0201702711/. It's probably a few years out of date now, but should still have a heap on the process of making purchases like this. That used to be one of my favourite reads... but then someone borrowed it at work (thanks whoever took it yeh bastard... hehe).
One problem with your setup is it does not provide enough disks for performance or caching to provide large amounts of I/O. ZFS might be good on big hardware, but I do not know that it would be practical on your rig.
Big vendors like EMC use excellent caching hardware and connectivity hardware that create a mesh of hundreds of disks using large amounts of memory within their storage arrays. Vendors like EMC spread data across tens and sometimes hundreds of disks along with the use of their caching mechanisms to prevent I/O bottlenecks depending on the application. While they have started using SATA disks, if you were to use disks that are too large in a rig like yours, you end up with lots of storage and too few disks across which to spread your I/O.
So, in summary, 12 sata disks might be good for a lab or development setup, but would not provide the performance of the "$50K" storage solutions. But then in most cases, if your lab does not match your production setup, what good is it?
FC actually pre-dates SATA They are blindingly fast, but Fibre Channel boxes are usually pretty pricey. You can do the same thing as I did, but you need a PCI controller board, (not mine which is PCI-X). I would check with the OpenSolaris forum, since they would have more experience with which Sparc drivers are available for you and which cards are best. Good Luck, that looks like a nice machine to play with.
A good video (in Real(tm) format) describing how ZFS is contructed and works is available from Sun's web site:
c enter.jsp
p df
http://www.sun.com/software/solaris/zfs_learning_
The slides that Bill Moore is showing are also available online:
http://www.sun.com/software/solaris/zfs_lc_preso.
It's difficult to advise without details of intended use, but generally I think it through like this:
Or, you could just buy this off ebay with 3TB of storage for under $2000. Radiator OS is linux based... http://www.infrant.com/products/products_details.p hp?name=ReadyNAS%20NVPlus
ZFS is a filesystem. It's a very impressive filesystem and does work very well in SAN environments.
What it won't do though is take your 3k toy and turn it into a SAN. Your box could maybe feed 2 applications in an enterprise environment. This isn't an issue with capacity but with performance.
A single SAN device serves 50 applications in my current environment and the plans for the new DC have a bigger HDS model serving around 100 apps.
Disks are cheap. The interconnect, processing, management tools and other bits and pieces are why we pay for a SAN. If you don't need that, then you are probably ok with a homebrew solution.
Another curious idea I have, though... is what happens if you use SATA-II disks, but only use 1/2 of the disk (the half closest to the outer tracks). A 500GB SATA-II disk would yield 250GB if you only used half the disk, but since your seek times are now cut in half, how does the performance stack up against a similarly-sized ~300GB 10K SCSI/FC/SAS disk? If they're similar (would really like to see this proven out...), one could make the argument that you can get by with lots of inexpensive SATA disks, but only half of them in use, at least during peak usage times. The other half of the disk could be reserved for off-peak bulk I/O, such as backups.
the real at&t mix
It's a decent idea, but gigabit ethernet isn't a fat enough pipe to keep up with SATA. I think you need to examine how many users it will serve and what the throughput requirements are for each of the users. If you have only a few users on a relatively small LAN, I would purchase fibre channel cards instead of gigabit ethernet. Then again, if it is for a large office environment, you're stuck with gigabit or 100 megabit ethernet like the rest of us cubicle-inhabiting suckers.
Forgive my ignorance, but how do I use AoE to provide high speed block devices to Windows systems?
I don't want to export them by CIFS or NFS because it is too slow. I want direct access to the devices from Windows boxes. iSCSI gives me this because ZFS (or Linux for that matter) can export iSCSI and Windows can "import" it as a block device.
The description of AoE sounds like it may be faster and easier than iSCSI, but how do I access it from a Windows machine?
- For the complete works of Shakespeare: cat
Using ZFS or even a big cheap RAID array as a backup or archiving target addresses short term issues but not long-term safety of archival data.
For the long term you have the issues of scaling and data migration of your archive. You need to minimize the chance of losing data due to hardware, software or human failures even as your storage scales up exponentially with time. You need to be able to incrementally/non-disruptively replace your backup disks and servers and networks as hardware becomes old and/or obsolete. You need to maintain accessibility and robustness of old data by migrating old data to new media. Businesses also need to worry about regulatory and best-practices enforcement of retention policies: some data you're required to guarantee won't get modified or deleted for mandated periods of time (and you must be able to prove that it hasn't been modified). There are commercial "archiving on disk" solutions that "solve" all of these problems (including eliminating risks due to the corruptible and failure-prone human components), but they involve a hell of a lot more than than ZFS and HA Linux!
The most obvious why this is no threat to real SANs is that you have a single cpu and single motherboard. If your board dies so does your access to the data.
It's simplistic to think about SAN and NAS as simply a capacity. You pay out the nose for the big SANs because you want high availibility, not just redundancy. As cute as this device is, it's not that.
I've always wondered why some people think that using 1U or 2U c chassis instead of normal PC cases is the only way to go if you want to look professional. Even places like ProCurve have stacks of average looking PC's.
Here's a picture just to prove it.
Add a cheap PCI IDE card and forget FC. It is highly unlikely you need the throughput.
An Education is the Font of All Liberty
And what happens when the RAID controller fails and corrupts all of your drives?
Because I've seen that happen more than once.
I'm not saying the more expensive solution is better. I'm just saying that in my personal experience I've seen *more* data destroyed from RAID controller failure than from hard drive failure. I would love to find out the solution to that one.
I do not claim to be a hardware expert or system administrator, so there may be a well known solution (don't buy 'brand X' RAID controllers). I just don't happen to know it.
I worked at ATMEL many years ago in their EPROM division. I had an up close and personal view of the screening flows, both Military and otherwise. Let's put aside the issue of Military screening, which is extensive and costly. You can't make very much out of Military grade ICs, because there are not very many available.
The difference between commercial and industrial parts is one of operating temperature, not quality. (In point of fact, there was no actual difference in the screening or handling.) The quality standards for both parts were the same - the goal was always zero defects. I spent weeks weeding out a problem with a 50 ppm failure rate that was slipping through our screening, and everyone was damned happy when I fixed it.
There's no reason to expect a correlation between maximum operating temperature and quality. A part might run too slow at elevated temperature to pass, but this will usually happen for process variation reasons that do not affect the expect lifetime of the part.
Any part coming from a reputable IC manufacturer should have the same level of quality, regardless of the rating.
Now, that being said, there is a very serious quality issue that an OEM does need to address, and that's counterfeit parts. If an OEM is not careful about where their parts come from, or buys them cheap and looks the other way, then there quality will obviously suffer. But this isn't so much a commercial versus industrial quality; it's about honest versus dishonest business practices.
It's not wasting time, I'm educating myself.
www.coraid.com
Coraid Storage Products connect to the network using standard Ethernet. With EtherDrive Storage you can create a shared storage system from a few Terabytes to multiple Petabytes for under $0.70 per Gigabyte.
O this learning! What a thing it is - William Shakespeare
gzip compression is available in ZFS in addition to the standard.
I've found that compression sometimes makes for faster writes.
You can pick up those 750GB Seagate SATA drives for about $200 each now...
Well, there's this.
try { do() || do_not(); } catch (JediException err) { yoda(err); }
We're going with an apple xsan + infortrend's eonstor solution. 2 x apple xserves running xsan, one qlogic sanbox (with sfp's), 2 x 4gb FC HBAs for the xserves, and 2 x 16 bay infortrend eonstors supporting 4GB FC with 750GB SATAII enterprise class seagates (single RAID controller, redundant PSU's) all for $38K. Unfortunately infortrend has not yet qualified the 1TB drives, otherwise we would have saved even more money. If you have a bigger budget you can get eonstors with redundant RAID controllers.
The nice thing about xsan is that we can keep adding eonstors to the racks or use larger drives in the eonstors and increase our storage. This is great since we're growing at about 2-3TB yearly.
not really. For the most part, these solutions are going to be used by different users. SAN/NAS is for enterprise users who have specific needs that are best met by fiber attached SAN on caching arrays; major enterprise entities -- very large corporations with complicated storage needs and thousands of servers. The sort of solution we are discussing here is for the person who isn't reliant on through put or the kind of caching and redundancy that an EMC or other brand name high end san solution might offer; consumer level or small business level purchasers with straightforward storage needs that reside on one or a few computers. There's a middle group that has to more carefully balance the cost/benefit analysis when choosing a storage solution. Do they require fiber throughput? Is price a more important factor? The large SAN products will likely lose some of these mid-tier clients, but giant corporations require support and indemnification that low cost systems such as the one in the article will never be able to provide.
Whether the thing you've described will meet your needs really depends on what you're looking for. Unfortunately, you've described the gizmo, but not what you need it to do.
That being said, here are a few things you need to be aware of:
1st problem:
From the Lime Technology web site, their MD-12000 "Unlike other RAID systems, however, user data is not striped across the data disk drives. Instead, each data disk is formatted normally with it's own file system." What this means is that it's going to be slow. SATA/IDE drives are slow to begin with, but without the striping it's going to be individual drive with an external connection slow. This also raises the question of how they're doing RAID5, since that requires multiple drives in order to hold the parity so that if one fails the data can be recreated when it is replaced.
2nd problem:
There is no mention of data cache anywhere in the technical information about the MD-12000. Without that, the seek/write times are going to be awful.
3rd problem:
The Lime Technology web site says that both the IDE and SATA verions are "temporarily out of stock". You can't even get one.
4th problem:
No spare drives. If you lose a drive, you need to get a replacement before you have protection again. RAID5 doesn't give you any protection at all from multiple drive failures, and yes it does happen. More frequently than you know. Especially with IDE/SATA drives. They don't have a particularly long lifespan, and years of experience with big storage has also shown me that you lose drives at spinup/spin down. Which means the age of your drives will be essentially the same and the chance to lose more than one is pretty good.
That all being said, if what you want is something inexpensive to attach to your desktop that will hold lots of data that isn't really important, needed by lots of people, and you don't need really fast access to, it sounds like a good deal. But once you start having to share that data with others from the same storage box, and need to get faster access, this thing isn't going to give you performance. It's never going to give you a really high amount of reliability. Not high enough to be putting critical data on it.
"Suppose you were an idiot..... And suppose you were a member of Congress... But I repeate myself."
In all the RAIDs I've seen all the disks must have the same sizes, which is a big expense when it's just for your personal photo storage needs. I'd like to see some kind of RAID that can take anything, just adding to it regularly to increase either storage size or redundancy or both in a controlled manner.
The reason is that I purchase a new drive every year or so, so I currently have 60G, 120Gb, 200Gb, 250Gb, 400Gb, 500Gb... Currently they are all mounted in different ways, but I'd love to have an enclosure where I can just add a new bigger disk and remove the oldest and smallest and keep going. Bonus point if I can plug it into the RJ45 of my adsl/wifi router or the USB of my laptop.
Does such a thing exist, it's exactly what I want ? Thanks
Non-Linux Penguins ?
Old guys like me may remember when National Semiconductor was, if I recall rightly, fined for faking test records in the same year they won an award for the reliability in the field of their military products. Or the discovery in the early 90s that volume produced Japanese semiconductors were far more reliable than many JAN devices. There is just no substitute for having to manufacture in volume in a competitive world.
In semiconductors, the downside is that things that produce higher reliability like thicker oxide and bigger anti-static diodes also slow down clocks. You would think that, for a really reliable disk array, you would a less than state of the art system running conservatively. I guess that this is a case where having a great deal of practical and experimental experience is the best recipe for success, and perhaps this is where SAN manufacturers shine.
Pining for the fjords
AOL has data centers full of mid-tower PCs. Say what you will about AOL itself, but they never go down.
"Obsolete" isn't a verb. Thank you.
-----
Sorry, I'm only a 1336 h4x0r.
I'm familiar with SATA and IDE...but, the FC ones are new to me..
;-)
Just a brief summary:
-- SATA refers to the new Serial ATA.
-- ATA or PATA refers to the older "Parallel" ATA. (ATA dates back to IBM PC AT and refers to that machine's AT Attachment interface.)
-- IDE refers to any drive (ATA, SCSI...) with integrated drive electronics, that is, everything that has come after the ancient dumb drives that required a model-specific controller on the motherboard. In other words, not a very useful term anymore.
-- SCSI refers to Small Computer System Interface; funny how it's the one used in the bigger iron. Beats the pants out of ATA when handling multiple daisy-chained drives; SATA is catching up in handling multiple drives. SCSI also has parallel interface and cabling.
-- SAS refers to Serially Attached SCSI (some inspiration from SATA perhaps?).
-- FC refers to Fibre Channel, a SCSI-like very fast interconnect type and interface protocol; often (but not always) uses optical cabling.
-- iSCSI refers to SCSI over Ethernet (thus it could be "SCSIoIP"...).
But I never understood the difference between a SAN and a NAS when the configuration gains any complexity beyond a textbook example. You can have a SAN with many NAS boxes, or you can have NAS with multiple SANs, sooo...
1TB is available in a single disk now, for something like $350. You can get desktop or rack cases that will accomodate more than four drives for less than $250. (Many years ago now, I bought a 4U aluminum tower case that could be loaded - tightly - with as many as nine hard disks if you had no optical drives, for $169 with no power supply. They are now cheaper.) You can also put disks in firewire enclosures and attach them to a hub, which will provide you some additional redundancy at the cost of adding clutter. Or you could buy Apple's RAID enclosures or something, and build your own filer to put them near on your rack, if you need a larger number of disks.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Actually, I don't quite understand the whole digital thing - JPEG, RAW, storage, etc....
I was curious about what you said so I looked in my folders where I store my digital photos. I have a 5 mp KODAK C533 and the typical jpeg file it makes is anywhere from 1.2 to 1.25 MB - right along with what you said. I guess the discrepancy from what you said is other stuff that the camera puts in the file?
Oh well, I'm still learning. Thanks for the lesson! You may have saved me some $$$ down the line!
I prefer Flambe as apposed flamebait.
You can put a few TB of storage into a single box with ZFS, but then you have a single box with a few TB of drives in them. ZFS replaces RAID.
SAN and NAS is about using networks for interconnecting storage and CPUs. I don't see what ZFS has to do with that at all, and ZFS probably can't even be used at all over some network-attached drives.
Don't forget that 80% of businesses in N. America are small businesses - under 500 people. And the majority of those businesses are under 75 people.
The whole point of this whole discussion is that relatively inexpensive NAS solutions are becoming a huge competitor to traditional SAN setups. Yes, FC/SCSI are likely more reliable than a plain-jane SATA setup, but often the cost just isn't justified in a smaller business.
I do IT consulting for the SMB market in Canada - companies ranging from 50 to 500 employees, and in many cases (not all) $20k - $50k is a lot of $$ to shell out for a couple TB of storage. Granted, the performance of a SATA NAS product isn't on par with a FC SAN product, but all things considered, for 10% the cost, you can have a decently reliable NAS product in place, and considering how cheap SATA drives are, you can have half a dozen spares sitting there, ready to go, for only a couple hundred bucks.
It all depends on the businesses needs at the end of the day. If a business can afford to spend $50,000 on a FC/SAN product, they're probably in a situation where they can't afford not to.
You've got a point; all this is possible w/out ZFS. But ZFS is very cool and makes it easier. ZFS is funny in that it is both amazingly revolutionary, while simultaneously an unimpressive incremental advance. It all depends on how you look at it.
Depending on the scale and application of the fileserver, generic OS provided software RAID will probably be more robust/recoverable than hardware RAID, without an unacceptable degradation in performance. I have, for example, an Adaptec 2410SA RAID adapter. The performance is not compelling, but the deal breaker for me is the card had on occasion, lost it's mind on unclean system reset. The drives were out of the array and that adapter offered no 'put together and assume it was a RAID-5 with this layout before, just sync the parity', so the data was non-trivially unavailable. With linux software RAID in particular (and the one time with Windows software RAID that I had to maintain as well), the tools in terms of what conditions can be recovered from and how it is to do are pretty smart and flexible. Both architectures I saw systems that decided (wrongly so) to put multiple drives on a single IDE channel and wrap in software RAID. A drive would effectively knock out accessibility for the other and the array would lose two members. For many hardware controllers, that would be 'the end'. Software RAID I was able to say 'assemble this array and assume I know what I'm talking about', and recover data without resorting to other backups which were inconvenient, or non-existant.
Some situations, you have particular hard throughput requirements you can quantify, in which case you may have little choice. Keep in mind if just doing simple file storage sharing (NFS/SMB) primarily over a single gigabit link or worse, you have really nothing much to gain from a high-performance internal subsystem. If you have local-storage for a high TPC database system or some disk intensive modeling or something, that's where you really start thinking about hardware controllers, IMHO,
XML is like violence. If it doesn't solve the problem, use more.
A SAN is not a host. It presents itself to a host machine as native storage in the form of raid groups/Luns, and/or raw storage. Access controls related to end users are done by the host OS, not the SAN, the SAN has no concept of file locking either, this is accomplished at the OS level on the host as well. Although the SAN does provide access controls for which host OS can connect to it. A NAS is the storage and some type of OS supplying network shares to a host. There are many tools that can make a NAS appear to a host as a native file system as well which kind of blurs the lines.
In really really simple terms, a SAN provides configurable disk space to a host, a NAS supplies file space and file serving to a host(s). Many storage solutions offer various functionality and can provide both NAS and SAN functionality at some level.
Don't forget to consider the need to maintain continuity of expertise for your solution. If you expect the same people to be around for years then a home-brew is more tenable. If you will have a lot of turnover of your admin/technical staff then factor in the bother of either continually retraining people or (more likely) them having to puzzle out the "old" system when they inherit it.
Avoiding single point of failure is about more than the cables between systems. Most architected SAN solutions are paranoid about even solid-state component failure. I.e. a RAID controller determines it has bad cache memory. In a typical SAN architecture, the controller notes the presence of a peer controller, throws up its hands, says replace me, and the other controller takes over, while the other controller is presumably hot swapped. If your controller with direct connected drives fails, well, that was your single point of failure. Your server suffers a catastrophic failure (PCI fault, processor fault, uncorrectable ECC), you have lost access to your storage. If you implement anything fancier (i.e. with multiple systems, talking to multiple controllers each with multiple paths to each individual drive), then by definition it becomes a SAN. Feel free to replace the FC fabric with iSCSI, or whatever commonly supported block-device-oriented strategy you want, it's still a SAN. Feel free to run ZFS on top of a SAN, though perhaps only in an active-passive way.
BTW, in many many ungodly amounts of explicit storage testing for data miscompare situations (why you would need above-block-layer checksuming), I have yet to see a single instance not caught by individual hard drive ECC mechanisms. I consider ZFS' checksumming nifty and good for demos involving dd and random data with an array member, but in real-world situations, it is largely redundant. Checksumming is done in lower levels transparently to the user, it's more of an assumed must-have than a 'hey look, we checksum!'. The checksumming would only catch errors that probably are obvious and caught before they ever reach ZFS. Now one thing I would like to see something do is upon seeing a bad block reported from a lower level, automatically go back and attempt to write back that one block of data based on other array member data. Bad block reallocation is nice for writes, but I have seen one array where by the time one bad, unrecoverable block was detected, other members of the array also had unrecoverable blocks in other places. All the data was still there, but the array wanted N-1 100% good members for the entire data set, instead of N-1 100% good members on a block-by-block basis. If instead of kicking out of the array, it would have rewritten the data it could then interpolate from other members, just a disaster would be averted (just monitor the SMART reported bad block reallocation count.)
XML is like violence. If it doesn't solve the problem, use more.
But it has been my experience that selling management types on anything where the word "cheap" is used or anything that is built in house is nearly impossible.
I killed da wabbit -Elmer Fudd
No, ZFS doesn't replace a SAN, but ZFS does checksums on every block in-kernel, so you're sure that what you wrote is indeed what you've read.
The performance will be slightly degraded (as opposed to other filesystems), but why wouldn't you want this feature on a SAN to guard against silent firmware defects or media errors?
o That is only a SINGLE CHANNEL IDE card (2 drives max)
? id=31c ontrollers/m/19271025
o It's a bit pricey compared to a Silicon Image IDE-133 RAID-capable card:
http://www.siliconimage.com/products/product.aspx
http://computers.pricegrabber.com/storage-device-
o IDE is $dying. SATA and other tech is the way to plan for $future, and believe me I am currently fairly heavily invested in IDE drives.
.
== WolfriderV6 == I'm willing to admit that *I just might* be wrong... Are you??
Slashdot - News for Herds. Stuff that Splatters.
I tried to crudely calculate a RAID5's MTTF recently, and even if I assume, the drive-manufacturers are exaggerating their drives' MTTFs ten times, I'm still getting MTTF of an array counted in centuries — assuming a failed-drive is replaced within a couple of days.
Here are my attempts (the text is the Gnuplot script, which produces the graphics), what do your company's experts say?
Even if my calculations are wrong, I suspect, the a failure of another disk, while the RAID is recovering from an earlier disk-failure is so improbable (even if the RAID spans dozens of drives), no efforts to reduce that already minuscule risk can possibly be justified. The companies peddling such reductions (RAID6 is how some solutions are called) are praying on their customers' being bad in Statistics...
Now, for RAIDs spanning many hundreds of drives — maybe...
In Soviet Washington the swamp drains you.
I would like to see some real world tests. Comparing an HP EVA8000 with say 32*146Gb FC-disks, against a ZFC with a decent hardware, like SCSI-Disks, GBIT-network, and everything as redundant as it can be.
;)
Run 10-15 servers against it with "enterprise-like-load" as Exchange 2003+Front-end, 2*DC's 1-2 App-servers, some fileservers and a clusterd MSSQL2005 Solution, maybe a CRM system and a ERP-system. We can even add some spice by adding a VMWARE ESX Server.
Let it run for some months. Which one causes most problems? Which one needs least hands-on-administration. How many severity 1 errors where the direct root cause was failed disk-communiation where there? (NOW don't forget 90% of the enterprises uses Windows-servers so this is a relevant so please dont begin to moan about GNU/Linux this and BSD that.)
What you all people seems to forget is the financial part of a failure. If a server breaks down for an e-commerce vendor, they lose real hard money. If the databases used for a research laboratory fails and 100's of researches cant work, how much do that cost? those money you saved on the homebrewn Storage system is long gone. Thats why enterprises tends to go for the more expensive and proven reliable stuff. (and when it comes to FDA-regulated industry, well don't get me started
But on the other hand, people who buy proven saftey tends to get out unharmed more often. Just why do you think people buy Volvo and not an obscure chinese brand?
"Never EVER mess with a jumper you don't know about, even if it's labeled 'sex and free beer'." - Dave Haynie
Now you have 47k to blow on a a team/office party! Remember, what was budgeted must be spent :)
True story:
Two years ago next month, a clumsy plumber got a propane torch too close to a sprinkler head with the expected consequences: LOTS of water took the path of least resistance, where it finally filtered it's way into the basement data center, coming out right on top of our SAN.
Obviously, it didn't survive.
"I don't know, therefore Aliens" Wafflebox1
Everyone things their data is important, but it's up to management to decide what the value of each type of data is. Certainly, the critical financial data mention in one post here has a high value. However, how long does it have to remain online? Other data has less value, but some tangible value, such as old email. How long does it have to remain online vs. the convenience of having it at your fingertips?
Managers pull apart the poor sysadmin, but in the end you have to tell them that they have to decide the value of each type of data, and segregate it appropriately. I did this in a crappy company I used to work for that made codecs for cellphones and video games. Management had its head up its collective ass, so I just took control (something I shouldn't have to do on a sysadmin's salary with no stock options) and told them the cost of data on each storage solution. I left before the plan was fully implemented, but the ideas was to have important financial data and current software in development on the expensive commercial NAS, the audio files, email, and miscellaneous junk on the cheap Linux RAID box I build out of white-box parts. The only time the linux box had a problem was when the AC failed in the data closet (the NAS was located right near it, and I wonder if it would have puked a few moments after the Linux box panicked. Note that having reliable environmental controls is probably more important than choosing the expensive hardware). I left before the division of data was fully separated, but it seemed like it was going well. Every manager's ass seemed to be fully covered, and at a reasonable cost.
I have been using AoE (ATA over Ethernet) in heavy production use for 9 months and it ROCKS. Block device semantics over commodity gigabit ethernet is just so useful. I love being able to decouple my disk storage from my cpu power. Especially handy when using something like Xen so you can do live migration of domains. Linux really is getting a lot of high-end features that you used to only be able to find on mainframes.
I've heard so far four stories where Dell just weaseled theirselves out of the "warranty". I don't know if this is a common pattern, but I will not buy Dell and do not recommend it any more.
So you pay premium and possibly get nothing in return. Not a good deal.
Anonymous Coward writes IDE refers to any drive (ATA, SCSI...) with integrated drive electronics, that is, everything that has come after the ancient dumb drives that required a model-specific controller on the motherboard. In other words, not a very useful term anymore.
Well, actually, IDE's history is a bit different than that. IDE requires a host buss interface, but, yes, they do have their disk controllers built into the PCA attached to the disk mechanism.
Before Compaq and others developed the first IDE systems, hard drives usually had external controller boards that used low level commands. IDE standardized the host interface to disk storage at the driver level, and standardized the host buss/drive command set at that buss level.
And, it's not just disk drives that use the IDE stack. Other devices can be attached to the IDE buss, too.
SCSI drives require a SCSI host buss adapter with a dedicated processor and that adapter does the heavy lifting for disk access. IDE requires the host CPU to do a lot of processing, where SCSI does the majority of the work. This model was used for the FC technology. It, too, unloads the processing from the CPU.
SCSI/FC are preferred in the 'big iron' type of installations. IDE/ATA/SATA are fine on a dedicated NAS system. In effect, the CPU of the NAS motherboard is doing the work that is done on the host buss adapters in SCSI/FC.
At the drive mech level, FC is a copper interface. The design of the connector on the disk mech allows it to be plugged. This provides the ability to quickly replace failed drives. The drive mechs are aggregated into some type of array to provide protection from data loss. This array of drives is then attached to systems via fiber optic cabling.
You can simulate some, but not all, of the benefits of a FC/SCSI array using SATA technology. I don't know if the IDE drivers are being rewritten to use the multi-core processors yet, but that would help reduce some of the latency.
Short answer, if what the OP was aiming for is to get into a large disk array for cheap, trading some reliability and performance for low cost, the idea is a good one. I would be looking for a multi-core cpu in the motherboard and an OS that has parallel processing drivers for the IDE channel. Be sure that all the drives have plenty of cooling. Have a backup solution. Some day, this lash-up will give you heartache, but till then, you've saved money.
Can you tell I used to work in the disk storage business?
Good luck.
Best regards.
I would avoid that card. It's limited to striping or mirroring, for starters. It's also not true hardware raid and depends on the drivers to do all the raid work. You really do get what you pay for here. You also get very little notice when one drive starts going bad. You just start getting random system hangs.
He could. He wouldn't want to.
:P If you use Gigabit Ethernet, it's going to be plenty fast either way.
The reason is that by exporting directly into Windows, you lose the #1 biggest advantage of this setup, with is LVM. In the install docs, it says to create your initial LV's to be only slightly larger than you need them, so I have actually only used about 200GB of my 7.5TB array right now. The reason is that you can always easily grow an LV, but shrinking, though may work, runs the risk of data loss. Under windows, you would have to format to the size you're going to use and that's that. I guess you *could* try using Partition Magic or similar, but...um...no.
Karma: Chameleon (mostly due to the fact that you come and go).
The 280R's FC hard drives don't have to be hooked up using an external Fibrechannel enclosure. FC-AL drives are just SCSI hard drives that use an internal, arbitrated loop FC connector (usually fiber) to increase transfer speeds as opposed to a SCSI cable. I can't remember if the drives come with a FC-AL enclosure, or if they're just regular SCSI drives and the FC-AL is on the Sun's backplane...
BTW, if you get an external array, be sure to check SunSolve to see if it's compatible with the 280R, and which OPB firmware version you need to support it, they can be fickle...
Delay is preferable to error. (Thomas Jefferson)
Because of the long times it takes to move these files around, I think NFS or CIFS would be too slow. That's why I am interested in the ability of ZFS to easily export iSCSI targets. Some tests I read showed that ZFS exporting iSCSI is about 4 times faster than ZFS exporting NFS or CIFS.
A friend of mine runs a big system with ZFS and quad-bonded gigabit between the solaris backend and linux frontends. ZFS with compression *on* (faster than off) can saturate his network.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Where's the uncertainty? Sun fears Linux, and their programmers have already admitted this is why they deliberately made a GPL-incompatible license. Using their patent minefield to prohibit GPL implementations would be incredibly foolish if widespread use of ZFS were actually their goal.
That's nice except Jonathan Schwartz has indicated that OpenSolaris will go GPL3, assuming the final version of the license is OK.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
The online OED allegedly has the use as a transitive verb marked (obs).
e s/002222.html
Word nerds:
http://itre.cis.upenn.edu/~myl/languagelog/archiv
Sadly, the primary source (oed.com) is not available through bugmenot.
you can have my violent video games when you pry them from my cold, dead hands.
Prime UID Club
Document your concerns to your 'higher ups', and then freshen up your resume'. Your boss isn't going to want to take the heat when this system goes down, so BOHICA, you're going to get the blame. Start your job search now.
And, ask them how much will it cost the company when the system fails and data gets lost.
The good news; the job market got better lately, so you won't be out of work for long.
Best regards.
Backups? add a small autoloader (like http://www.sun.com/storagetek/tape_storage/tape_li braries/c2/) + software like bacula. Relying on raid disks alone is stupid.
Multipathing? You want multiple links to your storage especially if you're using cheap unreliable off-the-shelf parts
Scalability? How do you grow this contraption once your greedy users start sucking up every byte you have. And yes, they will.
Exercise caution when modding this message up: the author acts like a jerk when his karma is excellent.
For this reason it's a good idea to buy spare controllers and, if you can manage it:
Have a warm-spare machine using software RAID. It's slower but it also means you can get your data back even if all you have is a sufficient number of working disks, a box to hold them, and a standard PC.
The poster's proposal sounds like a nice NAS, and a lousy SAN. A NAS saves you money by keeping your important data in a centrally-managed system with automated backups so that lay users cannot easily lose it, and so that it can be shared easily between multiple systems. A SAN saves you money by packing a big chunk of high-performance storage behind extremely-high-redundancy hardware and provisioning dedicated slices out to your servers so that you don't need dual redundant RAID controllers with mirrored write caches in each box that's dealing with critical data. They solve different problems.
Since a NAS centrally manages all of its storage, it can take advantage of fancy filesystem tricks to boost performance on SATA drives with good sequential access speeds. When you're physically provisioning chunks of disk on a SAN, you just can't do that, and you really need the 15k RPM drives with the 3 ms seek times, since you'll have numerous independent operations on physically discontiguous sectors on the platter, if performance matters at all. This is why SANs have enormous write caches.
Of course, since power failures or controller failures can knock out a write cache, high-end SANs have dual controllers with mirrored battery-backed write caches. This is specialized hardware, not something you can hack together from parts on Newegg.
The poster's idea is interesting for solving certain problems, but I don't think it's a panacea.
There's no failure quite as dissatisfying as a complete and total solution to the wrong problem.
... because this is essentially what the manufacturers of this type of system do (modulo file system, volume manager, specialized networking (perhaps), choice of base OS, and nifty cases). The main differences are this:
(1) The manufacturer can get hardware much cheaper than you. This doesn't make much difference in cost, because all of it goes to the profit margin. What it does buy you is that the manufacturer probably has a stock of these things that he can shove in if something breaks. At worst, he can have an entire working system delivered to your door before you could even start rustling up the parts for a replacement.
(2) The manufacturer has a service staff. What happens if you leave the company? What happens if you want to take a vacation? Someone else has to fix your cobbled up server. It's more likely that they'll screw up the configuration or break the thing. Un less you want to give 24/7 support.
(3) The manufacturer has better management software than your set of cobbled up scripts or web pages. At least it's been tested more thoroughly (probably).
(4) The hardware and software configurations are actually tested. Not that you won't do a few tests, but it's fairly likely that you won't do the burn-in testing that these manufacturers do. Nor can you guarantee appropriate SLAs.
So, yeah. You do pay more to these people. There's a reason. If I were your boss, I wouldn't let you build some wonk box. Maybe yours will. In the final analysis, you're just making more work for yourself. Besides, if you're so good at putting together these kinds of system, why aren't you out there in the marketplace competing with these high price suppliers? Don't tell me - it's just altruism for your boss...
That is all.
Thanks!
I prefer Flambe as apposed flamebait.
you have a few choices to make before buying a storage solution, here are your basic options:
storage capacity - how many terabytes do you want in your volume
reliability - you can get this through a few very reliable drives, or through many cheap drives
performance - ops/rate and streaming i/o performance
storage density - do you want a rack full of equipment? or a single small chassis
price - how much do you want to spend. (total cost of ownership)
most of these things compete with one another. you can maximize the first four and the price will be astronomically high. you can give up a few things (usually the density) and lower the price.
Anyone can build a cheap multi-terabyte array. but it will likely not be high performance or extremely reliable.
with things like ZFS, that solves a problem I didn't list. features of the filesystem that improve scalability, add features (like snapshots and replication), and improve the administration and maintainability of the system. and the flexibility grants you some limited reliability improvements, but good drives and lots of them is still the only real solution for reliability.
“Common sense is not so common.” — Voltaire
I smiled when I read this question as to the viability of the lower cost storage solution and how it compares to an enterprise-level solution such as that offered by EMC, NetApp, Hitachi, etc. It reminded me of a sysadmin consulting gig I was introduced to a few years ago. An energy consulting company of approximately 50 employees had a small data center that they wanted to contract out for IT support. Up until this point, the CFO had been designing and building the systems for everyone. When I walked into the data closet, he had a stack of various Dell laptops in various precarious positions in a single 2-post telecom rack. This was his server farm that they were using to perform careful analytical data of power plants, as well as store business critical data. This was the server farm. They had found that they ran out of storage rather quickly on each "laptop server", so of course there were various USB hard drives hanging off all over the place. When I inquired as to the rationale of this type of setup, he demonstrated a sense of proud accomplishment that he had solved a server consolidation issue on the cheap by just using people's old laptops and re-deploying them as servers. He didn't want the burden of those large 2U+ servers. Sure, many of the laptops had cracked screens, keyboards that didn't work locally, or just looked severely depressed, but I quickly gained the sense that it was not worth arguing with this person as to why HP or hell, even cheap Dell servers might be better.
While I am sure the long term laptop "server" maintenance would have kept be quite busy, I passed on the consulting gig.
Take ZFS out of the picture and you just need to use a hardware raid controller or a block level RAID
It's odd that so few posters have mentioned what, to me, is the #1 advantage of ZFS: end-to-end checksumming (combined with COW, redundancy, etc) guarantees your data in ways that no other RAID solution can.
you had me at #!
Google "end to end checksumming zfs" and READ ON.
you had me at #!
It is a common error for people to think that storage is storage, but there are quite a few differences between the configuration you spec'd and a decent modular fibre-channel array. Before I ramble on I would like to add that I have that exact case, stuffed full of 400GB SATA drives at home and it's *great* (although I am currently using LVM and ext3). I will also add that all of my friends and colleagues who use, or are still testing ZFS have pretty much come to the same conclusion. It's fast and virtually indestructible. Everyone seems to love it so far. I am a storage admin by trade and manage nearly 4000 disks and 2 dozen arrays. Your low cost solution is a good one for non-mission critical applications and low to medium I/O demands. Here is what the expensive arrays offer in addition to what your solution provides: Hot swappable, redundant (RHS) power. RHS controllers, cache and NVRAM. 4 or even 8 loops (used in pairs) to provide two active/active 200GB paths to each disk, or 300GB for SAS. Large write cache. Most drives have 8MB or maybe 16MB cache but the array may provide anywhere from 2GB to 32GB (or more) of cache, which makes writes a *lot* faster. Intelligent pre-fetch algorithms that, combined with cache, can significantly improve read response times. Configuration tuning options. Try changing your LUN layout without downtime. Try changing your cache block size on the fly. Try turning read caching off for write intensive LUN's, turning read caching on for a few more LUN's that do sequential I/O, and bumping up the number of blocks prefetched on those LUN's...all on the fly without interruption of service. Try updating the firmware on your controllers without disruption to service. Try doing parity scrubs on your drives without disruption. Try getting more statistics from your storage than you can get with iostat or sar. I'll leave support out of the equation but it should be a consideration when designing a solution. And now for a few notes on the CoolerMaster appliance model. How many controllers will you need to accomodate 7TB raw? And how many PCI slots will that require? Be aware that if you decide to go for maximum density (probably 750GB 7200RPM) you may be asking for quite a few IOPS per spindle. Now how about the network side of the equation. How many gig ports are you going to need? Doesn't take much to snarf a 12MB/s pipe in the disk world. And how many PCI slots will be required for those NIC's? You may have a tough time finding a motherboard that can accomodate your slot requirements *and* have enough bus/processor power to handle all that I/O, logical device overhead and filesystem overhead. While it is valid to suggest distributing load across many of these units, you may be introducing significant management overhead. Probably better off with a small number of scalable boxes that can handle a lot of disks per box. Enough rambling. ZFS is showing significant promise. You question is a fair one, and your configuration is a good one for certain applications. But it is *very* important to compile all of your business requirements before designing a solution. And those of us who have been around for awhile have learned that sometimes the least expensive solution ends up being quite a bit more expensive than anticipated (in downtime, manhours and sometimes, actual data loss).
I'd skip hardware RAID altogether and just get the most basic SATA controllers I could find. Even fake RAID controllers use vendor-specific disk signatures, which means when the cheap controller dies and you find out the company went out of business or the model is discontinued after 18 months, your data is locked away in a weird illogical format.
On the other hand, with software RAID and a dumb controller, you can move your drive arrays anywhere and they will work. I personally use Promise SATA2 cards because I was able to score them cheaply from my supplier, but go with whatever's available. They all pretty much use the same garbage Silicon Image chips so there's hardly any difference between the OEM stuff and the pricey brand-name adapters.
-Billco, Fnarg.com
Disclaimer: I AM a Storage Engineer at one of the big 3 SAN/NAS houses.
Here's what you get:
Double Parity RAID instead of RAID-5. Thousands of times more resillient. Read the latest literature on disk failure rates on SATA drives and then try to sleep at night knowing you only have RAID-5.
Speed: Dedicated storage OSes are optimized for raw speed. For more info see: Standard Performance Evaluation Corporation
Expandability: How much downtime do you require to expand your homebrewed system? Most commercial storage systems require none.
Support: What happens WHEN you do have a multi-disk failure (notice I said WHEN and not IF)? We can recover most multi-disk failures without data loss. Can you do that?
Disaster recovery and business continuity: Big box storage systems are built around this. What are you going to tell your CIO/Insurance company?
Sure, we cost more.
But how much is your data worth?
Goofy, Geeky Gifts and More!
First off, every serious NAS/SAN vendor is going to have a snapshot solution. Here are a couple other features you might need: Automatic Replication, High-Availablity / Failover, Integrated Virus Scanning, Clustering. Many of the "exciting" features listed in Sun's press releases are not even vaguely revolutionary.
As the name "NetApp" may imply, NAS/SAN vendors often sell "appliances" that attempt to simplify many of the management concerns (e.g. monitoring, automation of backups, etc). How well they do this varies from vendor to vendor and based on what features you need.
Large NAS/SAN vendors have already verified that their product works with a number of 3rd-party apps and hardware. Will your old tape hardware work well with ZFS? Is SQL certified to run on ZFS? Will Sun's customer support help you if you do get things working? Likely not.
For all the business I've heard about ZFS being the "last word in file systems", the amount of actual performance data has been incredibly lacking. For example, most NFS products have published their SPEC numbers. Although these performance results are often gamed a bit, they're the current standard for NFS performance.
The performance numbers I have seen with ZFS so far are useless (e.g. see here for someone measuring how fast ZFS can write to RAM or here for someone getting the blazing throuput of 45 mb/s). Filesystem-based iSCSI solutions (as opposed to SANs) tend to have terrible performance (this includes some of NetApp's products), so I'm a bit dubious about claims that ZFS does iSCSI faster than it does NFS.
In addition to reliability features such as High-Availability and Disaster Recovery, how many enterprise production environments is ZFS actually running in? How many data corruption bugs are waiting to be ironed out? How mature are the repair tools (e.g. fsck)?
Also, enterprise NAS/SANs (e.g. those of NetApp) often have a nice feature where an operation is stable once the client receives an ACK. They get this by logging pending operations to non-volatile RAM. As far as I can tell this is not possible with ZFS, which means that your applications need to be aware that operations may need to be resent to the server after crash.
The submitter needs to check which of these things are important to him/her, and then decide if ZFS is suitable. For homes and small offices NetApp, EMC, and most other large storage vendors are likely overkill. For others, enterprise NAS/SAN may be the only option.
[posted anonymously as my employer might not be happy with my post]
At home I favor a dedicated machine over SAN/NAS due to lack of programmability and auto backup. With LINUX, I expect easy crontab backup schemes!
Try to NFS export a ZFS volume and see what happens when multiple apps accessing the volume perform what they believe to be relatively cheap NFS sync RPCs to get a guarantee that the data has been committed.
The ZFS volume will not return from the sync nfsd generates to implement the RPC on the host storage until the ZIL (ZFS Intent Log) has been flushed. This synchronizes operations that should be parallel and makes the ZIL a bottleneck.
There are evil hacks to make ZFS lie about the state of files on disk returning from sync when the IO has been received (written to the ZIL). This apparently creates a race as subsequent reads can return data that has not been affected by the data written to the ZIL.
Please tell me I am wrong. I'd love to try this again, but I just cannot suck up the performance penalty.
--- Nothing clever here: move along now...
But I never understood the difference between a SAN and a NAS when the configuration gains any complexity beyond a textbook example.
A NAS is something you access via the network (CIFS/SMB, NFS, etc).
A SAN is something you access as a block device (/dev/sda, etc).
As of June last year, when Eric Shrock added this entry to his blog: ZFS Hot Spares
We have a large (geographically replicated) Hitachi disk array (as well as many NetApp boxes), mostly it works very well indeed.
However 2-3 years ago we stumbled (very painfully!) across a firmware bug which took the primary Hitachi array down:
As we (i.e. the Hitachi service reps) were upgrading the mirrored cache, an error hit the active half, and it turned out that the firmware would always check the mirror (a very good idea, right?) before falling back on re-reading the disk(s). However, the firmware error handler which could have handled an error on the mirror copy as well (as long as the data wasn't dirty, of course), did not know how to handle a _missing_ copy, instead it blew away the entire array while crashing.
It took us three days to get everything back up, even though most of the critical systems were running off of the WAN backup copy after 2-3 hours.
Terje
PS. That particular firmware bug has of course been extinguished, but there's bound to be some more lurking around. Getting totally non-stop operation is a _hard_ problem!
"almost all programming can be viewed as an exercise in caching"
If all you want is cheap NAS, you can do the same thing using any number of the free NAS solutions out there. Just add lots of cheap SATA disks in whatever cheap JBODs you can get your hands on, and you are done.
To replace a modest percentage of the functionalities of a mature NAS solution like Netapp's ONTAP + WAFL, you would need to:
1. Install Solaris 10 on a extremely reliable platform
2. Configure your zpools/filesystems
3. Set your NFS attributes
4. Install and compile all dependencies for Samba to support LDAP, Winbind, Kerberos (SFW stuff doesn't cut it)
5. Configure winbind/kerberos/LDAP
6. Configure LDAP backend in Samba (hint: NFS)
7. Either extend AD schema/turn on POSIX attributes or use a metadirectory to store idmaps
Now throw manageability into this equation...considering half of the admins out there have problems getting past step 5 above with any kind of consistency, Netapps suddenly makes a lot of sense, especially to companies that are contractor-happy and not willing to maintain that level of in house expertise.
What about scalability as a NAS (keyword NAS)? We can always throw SC 3.2 into the mix right? Unfortunately, NFS/Samba can only be configured as failover, and not scalable services in a Solaris Cluster. And ZFS? it can't be globally mounted and can only be used locally or for failover through HAStorage+...by the time you get to this point, you won't be using SATA drives with cheap JBODs anyway.
Now what about using ZFS to replace a SAN? Again, no.
A SAN's main function is to provide hosts with uniformed access to storage devices in your environment, be it through FC/Infiniband fabrics or iSCSI. This in turn allows you to better manage many aspects in your storage environment, such as but not limited to storage allocation, host access control/security, availability, and virtualization.
ZFS, at its core, is a file system. Yes, its well integrated with volume management, NFS, and self-healing functions. But it is still just a file system. A file system's job is to provide access to files (yes I know you can set up raw volumes with ZFS).
If you really look at the big picture, a file system is just another layer of virtualization in the storage game...by their very nature, filesystems and SANs sit at different layers in the OSI model. Comparing ZFS to SANs, is somewhat akin to comparing FTP to IP.
One cannot replace the other, they serve different functions in your storage environment.
What are you going to offer us next? World peace?
With EMC I get full support for the storage solution. If a drive fails I don't have to diagnose it and I don't have to change it, all is done as part of the *service* they offer.
With your solution I have to waste my time diagnosing and replacing disks. With SAN solutions I do system administration and planning.
I do not want to be a disk replacing monkey (and when you have hundreds or thousends of computers, each with several disks, EMC is worth the price, you will have disks failing daily and the amount of time to deal with disk failures becomes substantial).
People that have not worked in big IT states forget that what is bought is not a machine but a service.
In other words, you are comparing apples and oranges.
IANAL but write like a drunk one.
System administrator or disk changer (or machine changer, your pick) monkey?
When you state comprises more than a few servers you want everything standarized and rationalized. This saves time and money.
IANAL but write like a drunk one.
You can replicate accross data centres....
IANAL but write like a drunk one.
You will never revisit 99% of tha information.
I keep my best (or more menaingful) pictures and throw all the rest away.
SInce I am not into design or something that needs access to a big archive of pictures, my bad photographs do not need to be stored forever.
As for music, todays software makes easy to check what one is actualy listening to, stuf I have never heard but that was dwonloaded or ripped because I could gets regularly removed, I only keep my fav music or stuff I go back to regularly.
As for movies, I have lets say 2 hours/day to watch TV and/or movies. That is no more than 8 movies a month or around 100 a year, and this without counting going to the cinema, etc. So I keep 100 movies at all time and once in a while I get rid of some. I just don't need them and most likely will never been watched again.
I happily survive with 160GB with plenty of space to spare.
IANAL but write like a drunk one.
It's been a long time, but the Windows 2000 server I used to admin back in the day, software raid was integrated into the 'disk management' MMC plugin. I believe last I checked on a non-'server' MS install (2000 workstation), software raid beyond simple mirroring was not available (at least I couldn't find the same way to enable it I did on the server OS).
This was rather vague, but I know MS does this as well (that's one of the installs where I had to rebuild the array from it thinking two drives went bad concurrently). I'm sure google can help in more detail (it's obviously been years since I had to be a Windows admin, never even touched Server 2003).
XML is like violence. If it doesn't solve the problem, use more.
Is people insisting on high-performance RAID knowing full well that the system will only ever do NFS/SMB serving, not running any more complex processes locally, and accessing the disks via a single gigabit link. Or assuming absolutely that a RAID controller that is true hardware is guaranteed to be faster. I have seen RAID controllers where, true, the processor usage/throughput was technically lower (meaning the already largely idle cpus were more idle), the hardware engine on the card would hit its cap in performance well before the processor would hit any cap. Basically, if your criteria is 'must be hardware RAID', without any specific metrics and without researching the quantified performance numbers for the cards being looked into, then I think software RAID is probably really the answer.
Adding to your point about differing sets of tools, if you have an 8-year old server the raid controller goes out on, your chances of finding a compatible controller that can understand and import that array is slim, so you could have to rely unnecessarily on your backups just because your hardware vendor doesn't support you anymore. Hardware RAID has a lot of disadvantages.
XML is like violence. If it doesn't solve the problem, use more.
.... that does not matter. What I don't want is something that brakes on me easily. Both methods achieve this.
IANAL but write like a drunk one.
All technology has hiccups when it is first introduced.
It is completely irrational to abandon a new technology that clearly has advantages on the strength of *one* bad experience.
IANAL but write like a drunk one.
Is that your only machine and alll your bussiness plan relies on it be up and running? Then yes, maybe you should plan for redundancy as you are saying.
Do you have 500 machines (or more) providing different services? THen you would be mad not to "outsource" the disk management to an specialist via SAN.
IANAL but write like a drunk one.
I will not point all the juicy ironies there, that is an exercise to the reader.
Anyway, conceding that there are no substantial differences between the different kind of disks when it comes to quality (yeah, right) you talk about predictive failure. Well I talk about a disk fucked.
Once it is fucked, I have hundreds of machines in which I need to do multiple activities.
Do I need to waste my time to change that disk, ensure it is mirrored, etc? No, I am a systems administrator, not a DRM (Disk Replacement Monkey). Big companies with hundreds of computers actually save money using a SAN, since specialist time can be focussed in higher level activities.
I do not want to have 3 copies of each important machine just to save $47000 per system (as if). The would just multiply the chances of something going wrong.
You truly believe that SAN providers are screwing over thei costumers (banks! the ones stingiest with money when you come to think about it) bu miserably fail to appreciate the environments and demands in which a SAN is a perfectly justifiable, cost efficient solution.
That Google developped their own SAN software (or equivalent) does not mean that SAN companies specialized in those services are redundnat.
IANAL but write like a drunk one.
You are not factoring his time and lack of service.
Risk is a cost that has to be covered, most of the people defending this simple solution are not factoring that hidden cost that has to be covered somehow if you are professional (spare parts, duplicate systems, downtime costs, etc).
Get all that factored and what looked like a good idea begins to stink a bit fishy. Unless you need a departamentla server with data that is not vital for your bussiness.
IANAL but write like a drunk one.
The weak link is going to be the enclosure. Does it have dual interfaces? Dual Power? In the disk tray to chassis connection reliable? Does hot-swap really work, every time? Can you get environmentals from the enclosure? What about firmware upgrades? There is no Hell quite like cheap-ass disk enclosure Hell.
And someone get back to me in 7 years with the track record of these 1TB SATA disks, when subjected to continuous, database-style random access activity.
From the Line Tech web page: "Dual 300W power supplies. If you have 6 or fewer hard drives installed, then only one power supply needs to be on." - Not really dual-power IMHO.
And ZFS is no panacea, although it's a great step forward in Linux-land, and as far as I know, you still can't create a bootable ZFS volume. (It's been a while since I messed with it, so this may be wrong.)
Should be fine for your home system, though. Buy a cheap AIT drive on Ebay and keep it backed up every once in a while, you should be fine.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
I don't use the RAID "feature" on the card; but it's a decent cheap IDE expander. Pretty much standardized on it because there is Linux support for it out of the box, and if I need 4 more IDE drives or an extra DVD burner in a system, it fits the bill.
I've even started stepping away from LVM with drive capacities getting larger these days. One disk in an LVM goes bad and you're fscked.
.
== WolfriderV6 == I'm willing to admit that *I just might* be wrong... Are you??
I'd love to hear more hardware details on that file server setup. I'm looking into doing the exact same thing, except I have *zero* experience with Solaris. So any pointers you have would be great, but mostly I'd like to know what motherboard/chipset you ended up with that was cheap and supported.
You don't want RAID capable controllers. That keeps ZFS from seeing the individual drives and prevents much of the magic (like self-healing). This all works best with JBOD (Just a Box Of Disks).
Another n00b who thinks they can build a data-center capably 5-9's setup using cheap stuff they strung together. I love you guys. Let me take you on a tour of my email-server setup where I can randomly yank out fiber or even flip off one of the 3510FC arrays in my group, and watch it all keep purring along with 70,000 users NOT EVEN NOTICING. You simply don't understand that saving a few bucks on a lashed-together setup can be MUCH less important than having it "JUST WORK". Can your solution survive a mid-plane board or a controller or PSU going PHTHTHT! Can it survive an admin oops? Mine can. It's massively inefficient to do a RAID 5+1+0 as far as money, but it's VERY efficient as far as not ever having to say "yeah this service will be down for 3 days while we figure out how to unscramble things and restore the data." Professionalism means never having to say you're sorry. That's why there will always be work for guys like me. Because one day your "miraculously cheap" setup will go FIZZLE and people will look around for a home for their services that is first and foremost RELIABLE not cheap.