Slashdot Mirror
Does ZFS Obsolete Expensive NAS/SANs?
hoggoth writes "As a common everyman who needs big, fast, reliable storage without a big budget, I have been following a number of emerging technologies and I think they have finally become usable in combination. Specifically, it appears to me that I can put together the little brother of a $50,000 NAS/SAN solution for under $3,000. Storage experts: please tell me why this is or isn't feasible." Read on for the details of this cheap storage solution.
Get a CoolerMaster Stacker enclosure like this one (just the hardware not the software) that can hold up to 12 SATA drives. Install OpenSolaris and create ZFS pools with RAID-Z for redundancy. Export some pools with Samba for use as a NAS. Export some pools with iSCSI for use as a SAN. Run it over Gigabit Ethernet. Fast, secure, reliable, easy to administer, and cheap. Usable from Windows, Mac, and Linux. As a bonus ZFS let's me create daily or hourly snapshots at almost no cost in disk space or time.
Total cost: 1.4 Terabytes: $2,000. 7.7 Terabytes: $4,200 (Just the cost of the enclosure and the drives). That's an order of magnitude less expensive than other solutions.
Add redundant power supplies, NIC cards, SATA cards, etc as your needs require.
Get a CoolerMaster Stacker enclosure like this one (just the hardware not the software) that can hold up to 12 SATA drives. Install OpenSolaris and create ZFS pools with RAID-Z for redundancy. Export some pools with Samba for use as a NAS. Export some pools with iSCSI for use as a SAN. Run it over Gigabit Ethernet. Fast, secure, reliable, easy to administer, and cheap. Usable from Windows, Mac, and Linux. As a bonus ZFS let's me create daily or hourly snapshots at almost no cost in disk space or time.
Total cost: 1.4 Terabytes: $2,000. 7.7 Terabytes: $4,200 (Just the cost of the enclosure and the drives). That's an order of magnitude less expensive than other solutions.
Add redundant power supplies, NIC cards, SATA cards, etc as your needs require.
Not enough specifics here. I am going to say do your thing. If it works, you're a hero and saved 47k. If it doesn't obfuscate and negotiate the 50k of storage down to 47k. Win for all.
Unless you would like to give more specifics. Cause I am going to say in 99% of cases where you want fast, reliable, and cheap storage you only get to pick two.
Porn jokes indeed aside. I may not be an "everyman", but I think I'm close enough. My desire for storage (though not yet in the terrabyte range) comes from my photography (no not porn...). I take a bunch of pictures, and well, because storage is cheap I leave them all at the original file size (which in this case is about 2-5 MB depending).
I don't have a proper video camera, but I'm sure that people who do, have even bigger storage requirements.
Not only that, what with all the music you can copy of a friends HD now, your storage just jumps a bit more! (I've got literally more then 10 gigabytes of music on my desktop HD. And I know people who have hundreds of CDs, so if they ripped all those, they would have much more...)
Added to all those movies you can either rip or download...
Chuck in a decent network, family and/or friends, and you can now stream all this stuff around to wherever you want it.
I'd say then, that the most common use of all this space, multimedia. Not sure who has terrabytes of multimedia though.
I wank in the shower.
150GB mp3s
80GB DVDs
120GB games
14GB/hr for DV editing
1 whole drive for OSes
RAID-5ed (1 parity drive)
So I'm up to 4 200gb drives right now, without even trying hard.
Soon I'm going to jump to 500GB drives, and I expect to be hitting their limits in a year or so.
Also, how the hell am I supposed to back up all this?! Incrementals would be 10gb+ / week
Sure, a good $20k NAS RAID does more. Question is, is it really needed?
I mean, you could deploy 10 times as many of these as you could your array, giving me 10 times the storage.
Depending on what you do, it has to hit the network sometime. For example - we have a big expensive SAN solution. What's it used for? As a massive shared drive.
For a fraction the price we could of put a similarly sized one of these in every organization. Sure, it wouldn't be able to serve quite as much data, but most of our stuff isn't accessed extremely often anyways.
I've noted before that in many cases it'd be cheaper for us to backup to IDE hard drives than to tape. The tape alone costs more per megabyte than a HD, and has slower transfer rates to boot. Our backup solution could be several of these, connected by fiber to another building(so we don't have to move them).
I don't read AC A human right
You're not trying hard enough ;)
I've got just over a terabyte of live storage around the house and I probably use about half of it - I have a couple of hundred gigs of video and about 60 gigs of music. I know of someone who is currently buying seven of Hitachi's new terabyte HDs for an in-home video streaming system, There's always someone who has a use for it.
Precisely. The question in the title is a little bit like asking "Will large PC clusters obsolete mainframes?" or "Will Web applications obsolete traditional GUI applications?" The answer is, as always, "It depends on what you use it for." For high-performance databases or a high-traffic Exchange server, these things may not work well.
I've seen plenty iSCSI of solutions coupled with NAS servers that get pretty good throughput in this price range that are already integrated and ready to go, but the bottom line is that if you want high-peformance, high-availability storage for I/O-intensive applications, you need a fiber SAN/NAS solution.
My blog
I agree. At my work, we have a SAN ... low-end frames (SATA) to mid-range (FC+SATA) to high-end frames (FC.) We put a front-end on the low-end and mid-range storage using a NAS, so you can still access using the storage fabric or over IP delivery. Having a SAN was a good idea for us, as it allowed us to centralize our storage provisioning.
I'm familiar with ZFS and the many cool features laid out in this Ask Slashdot. The simple answer is: ZFS isn't a good fit to replace expensive SAN/NASs. However, ZFS on a good server with good storage might be a way to replace an inexpensive SAN/NAS. Depending on your definition of "inexpensive." And if you don't mind the server being your single point of failure.
Show me how you build a Raid 50 of 32 sata or ide drives.
also show me a SINGLE sata or ide drive that can touch the data io rates of a u320 scsi drive with 15K spindle speeds.
Low end consumer drive cant do the high end stuff. Dont even try to convince anyone of this. guess what, those uses are not anywher near strange for big companies. witha giant SQL db you want... no you NEED the fastest drives you can get your hands on and that is SCSI or Fiberchannel.
Do not look at laser with remaining good eye.
Potentially it will obselete low-end NAS/SAN hardware (eg: Dell/EMC AX150i, StoreVault S500) in the next couple of years, for companies who are prepared to expend the additional people time in rolling their own and managing it (a not insignificant cost - easily making up $thousands or more a year). There's a lot of value in being able to take an array out of a box, plug it in, go to a web interface and click a few buttons, then forget it exists.
However, your DIY project isn't going to come close to the performance, reliability and scalability of even an off the shelf mid-range SAN/NAS device using FC drives, multiple redundant controllers and power supplies - even if the front end is iSCSI.
Not to mention the manageability and support aspects. When you're in a position to drop $50k on a storage solution, you're in a position to be losing major money when something breaks, which is where that 24x7x2hr support contract comes into play, and hunting around on forums or running down to the corner store for some hardware components just isn't an option.
ZFS also still has some reliability aspects to work out - eg: hot spares. Plus there isn't a non-OpenSolaris release that offers iSCSI target support yet AFAIK.
I've looked into this sort of thing myself, for both home and work - and while it's quite sufficient for my needs at home, IMHO it needs 1 - 2 years to mature before it's going to be a serious alternative in the low-end NAS space.
You did not read the reports out recently on drive reliability.
SCSI vs IDE is no issue. Newer technology drives are better than older drives is a bigger factor. So SATA lasts as long as SCSI. Performance wise for sequential access, a 500GB 7200RPM disk beats a 10000 RPM 72GB or 144 GB SCSI any day. For my laptop, my 160GB 5400 RPM disk is faster than the 7200 RPM 100GB disk.
The advantage of the SCSI disk is native command queuing. So that it can stop on the way from sector A to C and read sector B. But, this is also implemented in SATA-300 drives, so this advantage is also gone from SCSI.
I have been a big SCSI fanboy myself, but the magic has gone. SATA has flown past SCSI.
Comparing drive to drive, I agree with you; 15k wins.
However, the price point on 15k drives is such a comparison for a single drive vs multiple drives is reasonable. The basis is $$/GB, not drive A vs drive B.
Ask Google how they get their throughput on their terrabyte datasets. Hint: it's not due to 15k drives.
Anything is possible given time and money.
Again it depends.
At $3000 for everything it would be logical to stock a spare motherboard, Power supply, NIC, controller cards, and A few Hard drives plus a few hot spares.
With SMART it isn't hard and other sensor packages it wouldn't be hard to build in monitoring.
It could be done but it would be a project. So yes it could be done but for a business buying a SAN will be more logical if for no other reason than time. It would take time to set up a system. It could be cheaper and have just as little downtime as a commercial SAN but it could also be a huge time sink. For a small tech company it might be a great solution and a potental product. For a HUGE tech company it might be a good solution and a potental product. For anybody else it is a risk probably not worth taking.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
Unless you *need* the performance or it is part of a system that $VENDOR is supporting. I would *strongly* suggest against hardware RAID cards; especially the low end ones. If you use a pure software RAID in an operating system that makes the source code available - if it all goes wrong you can read the source and work out how to put the pieces back together. I've done it. It's not fun. At all. But it is possible. If the same thing happens with a hardware RAID (or heaven forbid, some of the modern win-RAID, propriatory software + undocumented hardware, low end offerings), then unless the 'recovery' tools you are given can do it; you're stuffed. Do not underestimate the value of having a recovery option for worst case scenarios.
From the ZFS FAQ:
Q: Suppose I have three E450s. Would ZFS allow me to integrate storage across all three boxes into one big "poor man's SAN"?
A: No, ZFS is a local filesystem (for the time being). To access storage attached to a different host, use NFS.
You can buy 10Gb ethernet to run iSCSI for less than FC.
I'm not sure where you're shopping, but I've only seen a handful of 10Gb ethernet switches, and they have all been dreadfully expensive. So factor in a pair of those, plus multiple 10Gb TOE cards, and I bet that your price isn't any cheaper than fiber. In fact, it's probably more expensive.
It makes huge sense to use the same technology for your LAN & SAN.
Not always, and I think that this tends to be overstated by the iSCSI proponents. This statement makes perfect sense on the surface, but when you look into the details it comes up short. It makes sense from a knowledge transfer standpoint to use the same technology. You can get by with a single person who understands TCP/IP networking instead of a person who does networking and a person who does SAN networking. But that person who handles the networking for the SAN usually also handles all of the zoning, space allocation, LUN carving, and everything else that comes with managing a SAN. Who does that now?
Then there's the whole issue of network segregation. Do you really want to put your SAN on the same switches are your desktop PCs or servers? Probably not. It doesn't make a lot of sense. Sure, you can VLAN it off, but those switches still have a finite amount of bandwidth available, and you don't want I/O intensive applications (like iSCSI SANs) eating up all of the bandwidth that users are accustomed to having. Then there's the speed issue. Do you buy a ton of 10Gb switches for all of your datacenter network, or just a few for the iSCSI SAN? Do you run iSCSI on 1Gb switches instead? What you're going to end up doing eventually is just buying dedicated networking hardware for the SAN and physically separating it from the rest of your network. So at that point, what advantages do you get with iSCSI over fiber?
Then you don't know what ZFS is.
Frag 'em all...
> Why not IBM's JFS? Or ReiserFS?
Because they are just filesystems. ZFS is also a volume manager.
> Or a CentOS based OS in place of Solaris?
Because CentOS doesn't have ZFS.
+4 Interesting. Awesome.
I worked at ATMEL many years ago in their EPROM division. I had an up close and personal view of the screening flows, both Military and otherwise. Let's put aside the issue of Military screening, which is extensive and costly. You can't make very much out of Military grade ICs, because there are not very many available.
The difference between commercial and industrial parts is one of operating temperature, not quality. (In point of fact, there was no actual difference in the screening or handling.) The quality standards for both parts were the same - the goal was always zero defects. I spent weeks weeding out a problem with a 50 ppm failure rate that was slipping through our screening, and everyone was damned happy when I fixed it.
There's no reason to expect a correlation between maximum operating temperature and quality. A part might run too slow at elevated temperature to pass, but this will usually happen for process variation reasons that do not affect the expect lifetime of the part.
Any part coming from a reputable IC manufacturer should have the same level of quality, regardless of the rating.
Now, that being said, there is a very serious quality issue that an OEM does need to address, and that's counterfeit parts. If an OEM is not careful about where their parts come from, or buys them cheap and looks the other way, then there quality will obviously suffer. But this isn't so much a commercial versus industrial quality; it's about honest versus dishonest business practices.
It's not wasting time, I'm educating myself.
ZFS will do some things the others file systems can't. First off it is "copy on write" and keeps a complete backwards version history so if a file is damaged, deleted or you just need to back out a change to a word processing document you can do that. Also ZFS moves both volume managment and raid into the file system. You can add and remove physical drives without stopping the system. And of course it is huge. A 128 bit file system can't ever be filled. (yes "never" do the math) It's also fast and maintains end to end checksum. Sun really has raised the bar here. That said this is not what a typical home user with only a hand full of disk drives and users needs.
Back to the question. Can it replace a SAN. Depends on the required performance. If you have 25 or 30 video edit workstations or a corporation with 5,000 desk tops it's hard to see how one Solaris server is going to work. You need something that can a lot of IO bandwidth.
Actually it is exactly what a home user needs:
ZFS loves cheap disks.
And with only two commands (zpool and zfs) and integrated web frontend, it's very simple to use.
We do that as a preventative measure for people that don't get support through us. We don't want anybody to assume that the file system is ready for a highly-available cluster without talking to us first. As with all file systems, there are trade-offs to using Starfish. We are being honest - the software is stable as far as we can tell, but it doesn't have a great deal of field use (it was released to the public in March 2006).
We must be especially careful with file systems - data is very important to people. If somebody uses our system and loses data, we can't fix that - not that it has ever happened. We put the beta message as a warning that people should talk to us before thinking about putting our system into production. After all - it doesn't have nearly the amount of testing behind it that EXT3 does. It is common for a file system to remain as a beta product for the first year or two.
Hmm... ZFS and Starfish aren't really meant to address the same storage problem. Take a bit of time and read through what ZFS does and what Starfish does. ZFS is a block-level file system. Starfish is a file-level distributed clustered storage system. Those are two very different things - at the end of the day they store files, but in very different ways and for very different purposes. Starfish can use ZFS as it's block-level file system... it can also use Reiser and EXT3.
Here are a couple of reasons to use Starfish (even though we think that ZFS is a fantastic solution for block-level file system problems):
However, this really isn't a "what is better, Starfish or ZFS?" discussion. You can have the best of both worlds: Starfish as the file-level network storage cloud using ZFS as the block-level file system.
-- manuManu Sporny (skype: msporny, twitter: manusporny, G+: +Manu Sporny)
Founder/CEO - Digital Bazaar, Inc.
Thanks for being a dick. Maybe a little explanation instead of your put-downs would have been more helpful.
This is why you replicate to another site :).
I haven't tried it personally, but it should work just fine with "zpool add " (might need a -f after add). RAID Z even copes well with different-sized disks, a configuration I was running for a short while while I waited for bigger disks to arrive. What you cannot do easily is decrease the size of a RAID Z. Currently there is no non-trivial way to permanently reduce the size of a RAID Z by removing a disk. Support for that is planned, however.
"Alcohol, Tobacco, Firearms, and Explosives" should be a convenience store, not a government agency.
In the development branch, yes. If you want a welltested ZFS, Solaris is it.