Does ZFS Obsolete Expensive NAS/SANs?

hoggoth writes "As a common everyman who needs big, fast, reliable storage without a big budget, I have been following a number of emerging technologies and I think they have finally become usable in combination. Specifically, it appears to me that I can put together the little brother of a $50,000 NAS/SAN solution for under $3,000. Storage experts: please tell me why this is or isn't feasible." Read on for the details of this cheap storage solution.

Get a CoolerMaster Stacker enclosure like this one (just the hardware not the software) that can hold up to 12 SATA drives. Install OpenSolaris and create ZFS pools with RAID-Z for redundancy. Export some pools with Samba for use as a NAS. Export some pools with iSCSI for use as a SAN. Run it over Gigabit Ethernet. Fast, secure, reliable, easy to administer, and cheap. Usable from Windows, Mac, and Linux. As a bonus ZFS let's me create daily or hourly snapshots at almost no cost in disk space or time.

Total cost: 1.4 Terabytes: $2,000. 7.7 Terabytes: $4,200 (Just the cost of the enclosure and the drives). That's an order of magnitude less expensive than other solutions.

Add redundant power supplies, NIC cards, SATA cards, etc as your needs require.

Comments

Specifics please.

[PowerEdge]

Not enough specifics here. I am going to say do your thing. If it works, you're a hero and saved 47k. If it doesn't obfuscate and negotiate the 50k of storage down to 47k. Win for all.

Unless you would like to give more specifics. Cause I am going to say in 99% of cases where you want fast, reliable, and cheap storage you only get to pick two.

Re:Specifics please.

[tgatliff]

"comsumer-grade hardware"???

Do you honestly believe the slogan of "business-grade"? Come on, let the marketing jargon go. Hardware designs are expensive, so rarely are there multiple designs. Sales guys are selling you additional support, but the hardware is rarely different. If it is, then the volume is not there, so the reliability is actually worse. Volume is the king of reliability. Reliability is always more dependent on the age of the design and its volume rather than the intended customer...

Re:Specifics please.

[Ngarrang]

Unless you would like to give more specifics. Cause I am going to say in 99% of cases where you want fast, reliable, and cheap storage you only get to pick two.

I disagree completely. Computer hardware is a commodity. The big box makers are afraid of this very kind of configuration which would blow them out of business if more people caught on to it. No, they use FUD to convince PHBs that because of the low cost, it cannot possibly be as good. Hot-swap and hot-spare are commodity technologies. But, please, feel free to continue the FUD, because it helps the bottom line.

Re:Specifics please.

[toleraen]

So "Mission Critical" is just a myth too, right?

Re:Specifics please.

[Score Whore]

No kidding. Without specific details there is no way to answer whether this is a good solution to his particular situation. However, even in the absence of details I can say this:

1) That case has twelve spindles. You aren't going to get the same performance from a dozen drives as you get from an hundred.

2) That system includes a small Celeron D processor with 512 MB RAM. You aren't going to get the same performance as you'll get from multiple dedicated RAID controllers with twenty+ gig of dedicated disk cache.

3) Your single gigabit ethernet interface won't even come close to the performance of the three or four (or ten) 2 gigabit fibre channel adapters involved in most SAN arrays.

4) Software iscsi initiators and targets aren't a replacement for dedicated adapters.

5) The Hitachi array at work never goes down. Ever. Not for expansions. Not for drive replacements. Not for relayouts. The reliability of a PC and the opportunity to do online maintenance won't approach that of a real array.

Don't get me wrong. That case makes me all tingly inside -- for personal use. But as a SAN replacement, fuck no. It's not the same thing. The original question just shows ignorance of what SANs are and the roles they fill in the data center.

As a workgroup storage solution for a handful of end users on their desktops, that solution probably may be a good fit. As a storage solution for ten (or two hundred) business critical server systems, no way.

Re:Specifics please.

[zerocool^]

Solly Chollie.

We have one of those "$50,000 SANs". Actually, with the expansion, ours cost closer to $110,000, but whatever. For one, it's got more drives (I think we're up to 30). For two, they're 10k RPM SCSI drives. For three, it's fibrechannel, and all the servers that it's connected to get full speed access to the drives. For four, we have a service level agreement with Dell/EMC (It's an EMC array, rebranded under Dell's name) that says if we have a dead drive, we get a dell service hobbit on site within 4 hours with a new drive. We also get high level support from them for things like firmware upgrades (we had to go through that recently to install the expansion).

The grandparent is exactly correct:
fast, reliable, and cheap storage - you only get to pick two.

We picked fast and reliable. You picked reliable and cheap. There's nothing wrong with either decision, but please don't tell me that a $3000 solution would have done everything that we needed.

You can't have all three.

~Wx

Re:Everyman?

[apathy maybe]

Porn jokes indeed aside. I may not be an "everyman", but I think I'm close enough. My desire for storage (though not yet in the terrabyte range) comes from my photography (no not porn...). I take a bunch of pictures, and well, because storage is cheap I leave them all at the original file size (which in this case is about 2-5 MB depending).

I don't have a proper video camera, but I'm sure that people who do, have even bigger storage requirements.

Not only that, what with all the music you can copy of a friends HD now, your storage just jumps a bit more! (I've got literally more then 10 gigabytes of music on my desktop HD. And I know people who have hundreds of CDs, so if they ripped all those, they would have much more...)

Added to all those movies you can either rip or download...
Chuck in a decent network, family and/or friends, and you can now stream all this stuff around to wherever you want it.

I'd say then, that the most common use of all this space, multimedia. Not sure who has terrabytes of multimedia though.

Re:Everyman?

[Baddas]

150GB mp3s
80GB DVDs
120GB games
14GB/hr for DV editing
1 whole drive for OSes
RAID-5ed (1 parity drive)

So I'm up to 4 200gb drives right now, without even trying hard.

Soon I'm going to jump to 500GB drives, and I expect to be hitting their limits in a year or so.

Also, how the hell am I supposed to back up all this?! Incrementals would be 10gb+ / week

Re:ok for low end, not for high

[morgan_greywolf]

Precisely. The question in the title is a little bit like asking "Will large PC clusters obsolete mainframes?" or "Will Web applications obsolete traditional GUI applications?" The answer is, as always, "It depends on what you use it for." For high-performance databases or a high-traffic Exchange server, these things may not work well.

I've seen plenty iSCSI of solutions coupled with NAS servers that get pretty good throughput in this price range that are already integrated and ready to go, but the bottom line is that if you want high-peformance, high-availability storage for I/O-intensive applications, you need a fiber SAN/NAS solution.

Re:ok for low end, not for high

[Jim Hall]

I agree. At my work, we have a SAN ... low-end frames (SATA) to mid-range (FC+SATA) to high-end frames (FC.) We put a front-end on the low-end and mid-range storage using a NAS, so you can still access using the storage fabric or over IP delivery. Having a SAN was a good idea for us, as it allowed us to centralize our storage provisioning.

I'm familiar with ZFS and the many cool features laid out in this Ask Slashdot. The simple answer is: ZFS isn't a good fit to replace expensive SAN/NASs. However, ZFS on a good server with good storage might be a way to replace an inexpensive SAN/NAS. Depending on your definition of "inexpensive." And if you don't mind the server being your single point of failure.

Re:ZFS

[chegosaurus]

> Why not IBM's JFS? Or ReiserFS?

Because they are just filesystems. ZFS is also a volume manager.

> Or a CentOS based OS in place of Solaris?

Because CentOS doesn't have ZFS.

+4 Interesting. Awesome.

Quality - industrial vs. commercial

[FuzzyDaddy]

The "consumer-grade" and "business-grade" are the same off the shelf stuff, but if you are getting business-grade stuff from a reputable vendor they QA the consumer-grade parts, throw out the bad ones, and stamp "business-grade" on the ones that survive.

I worked at ATMEL many years ago in their EPROM division. I had an up close and personal view of the screening flows, both Military and otherwise. Let's put aside the issue of Military screening, which is extensive and costly. You can't make very much out of Military grade ICs, because there are not very many available.

The difference between commercial and industrial parts is one of operating temperature, not quality. (In point of fact, there was no actual difference in the screening or handling.) The quality standards for both parts were the same - the goal was always zero defects. I spent weeks weeding out a problem with a 50 ppm failure rate that was slipping through our screening, and everyone was damned happy when I fixed it.

There's no reason to expect a correlation between maximum operating temperature and quality. A part might run too slow at elevated temperature to pass, but this will usually happen for process variation reasons that do not affect the expect lifetime of the part.

Any part coming from a reputable IC manufacturer should have the same level of quality, regardless of the rating.

Now, that being said, there is a very serious quality issue that an OEM does need to address, and that's counterfeit parts. If an OEM is not careful about where their parts come from, or buys them cheap and looks the other way, then there quality will obviously suffer. But this isn't so much a commercial versus industrial quality; it's about honest versus dishonest business practices.

Re:ZFS

[ChrisA90278]

ZFS will do some things the others file systems can't. First off it is "copy on write" and keeps a complete backwards version history so if a file is damaged, deleted or you just need to back out a change to a word processing document you can do that. Also ZFS moves both volume managment and raid into the file system. You can add and remove physical drives without stopping the system. And of course it is huge. A 128 bit file system can't ever be filled. (yes "never" do the math) It's also fast and maintains end to end checksum. Sun really has raised the bar here. That said this is not what a typical home user with only a hand full of disk drives and users needs.

Back to the question. Can it replace a SAN. Depends on the required performance. If you have 25 or 30 video edit workstations or a corporation with 5,000 desk tops it's hard to see how one Solaris server is going to work. You need something that can a lot of IO bandwidth.

Re:Have you tried out Starfish?

[msporny]

Really? Cause, your download page warns that it's a beta product and should not be used in production.

We do that as a preventative measure for people that don't get support through us. We don't want anybody to assume that the file system is ready for a highly-available cluster without talking to us first. As with all file systems, there are trade-offs to using Starfish. We are being honest - the software is stable as far as we can tell, but it doesn't have a great deal of field use (it was released to the public in March 2006).

We must be especially careful with file systems - data is very important to people. If somebody uses our system and loses data, we can't fix that - not that it has ever happened. We put the beta message as a warning that people should talk to us before thinking about putting our system into production. After all - it doesn't have nearly the amount of testing behind it that EXT3 does. It is common for a file system to remain as a beta product for the first year or two.

On the other hand, there stable ZFS support in OpenSolaris, (non-open) Solaris, and in a few months in FreeBSD, and since it's available fully free and open source, there are no necessary licensing costs. Are there any advantages of Starfish over ZFS?

Hmm... ZFS and Starfish aren't really meant to address the same storage problem. Take a bit of time and read through what ZFS does and what Starfish does. ZFS is a block-level file system. Starfish is a file-level distributed clustered storage system. Those are two very different things - at the end of the day they store files, but in very different ways and for very different purposes. Starfish can use ZFS as it's block-level file system... it can also use Reiser and EXT3.

Here are a couple of reasons to use Starfish (even though we think that ZFS is a fantastic solution for block-level file system problems):

• Starfish is a decentralized, multi-node fault tolerant file storage solution that provides N-way redundancy.
• Starfish is useful when you have a large number of nodes that need to access data in parallel. The ability to perform asynchronous parallel throughput is one of Starfish's biggest advantages.
• Starfish runs in userspace - which means that it is capable of using much smarter algorithms and databases to manage metadata and file placement. For example, a SQL database is used for file system metadata, which means that a variety of optimizations (such as creating metadata indices) to the file system can be performed at runtime.

However, this really isn't a "what is better, Starfish or ZFS?" discussion. You can have the best of both worlds: Starfish as the file-level network storage cloud using ZFS as the block-level file system.

-- manu