Open Source Highly Available Storage Solutions?

Gunfighter asks: "I run a small data center for one of my customers, but they're constantly filling up different hard drives on different servers and then shuffling the data back and forth. At their current level of business, they can't afford to invest in a Storage Area Network of any sort, so they want to spread the load of their data storage needs across their existing servers, like Google does. The only software packages I've found that do this seamlessly are Lustre and NFS. The problem with Lustre is that it has a single metadata server unless you configure fail-over, and NFS isn't redundant at all and can be a nightmare to manage. The only thing I've found that even comes close is Starfish. While it looks promising, I'm wondering if anyone else has found a reliable solution that is as easy to set up and manage? Eventually, they would like to be able to scale from their current storage usage levels (~2TB) to several hundred terabytes once the operation goes into full production."

Comments

Entry level SAN?

[lukas84]

When you're talking HA, you're always talking "big money". If you want a fully redundant infrastructure, you might have to start using commercial operating systems (like Novell Linux, RedHat, or even other Unix-based commercial OS like Solaris or AIX). The problem here is support. Full HA environments are incredibly complex, and you will need to make very, very sure that everything works well.

I wrongly implemented HA system will have less uptime than a 499US$ Dell with a single ATA drive.

Entry level SANs using iSCSI are available at quite affordable prices. Look at HPs and IBMs (e.G. the DS300). Even the entry models allow you to use MPIO.

Re:Entry level SAN?

[jabuzz]

Thing is they want to be able to go from 2TB to hundreds of TB and they cannot afford a SAN!!!

They need to accept that this ain't going to happen, and what they need to do is put in a solution for now and plan for a different solution when they go into production and presumably have the money.

However one has to wonder if there current storage requirements are a messily 2TB why the heck do they need more than one server, unless it is a second for failover.

Rethink your drink

[wcspxyx]

Before you go about deciding what file system you need, you need to spend some time thinking about what kinds of files your customers are storing. RDBMS data? Large graphics/audio/video files that rarely if ever change? Scanned documents? Large numbers of small files? Small numbers of large files? You get the idea.

Then you can start looking at solutions. 'Optimal File System' can mean many things to many people, and everyone here is going to have a different viewpoint. You need to decide what features of a file system makes it optimal for you. Then you can go looking for a solution.