Slashdot Mirror

Except when they do support redundancy:

http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Creating_Replicated_Volumes - Replicated volumes replicate files throughout the bricks in the volume. You can use replicated volumes in environments where high-availability and high-reliability are critical.

RAID is still NOT A BACKUP!

I have a 500 node replicated filesystem... and I just overwrote the wrong file, or a virus infected a file, or the file got corrupted...

The good news is my 500 replicated nodes are all consistent. The bad news is... wheres my fucking file!

Except when they do support redundancy:

http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Creating_Replicated_Volumes - Replicated volumes replicate files throughout the bricks in the volume. You can use replicated volumes in environments where high-availability and high-reliability are critical.

http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Creating_Distributed_Striped_Volumes - Distributed striped volumes stripe data across two or more nodes in the cluster. For best results, you should use distributed striped volumes where the requirement is to scale storage and in high concurrency environments accessing very large files is critical.

http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Creating_Distributed_Replicated_Volumes - Distributes files across replicated bricks in the volume. You can use distributed replicated volumes in environments where the requirement is to scale storage and high-reliability is critical. Distributed replicated volumes also offer improved read performance in most environments.

Except when they do support redundancy:

http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Creating_Replicated_Volumes - Replicated volumes replicate files throughout the bricks in the volume. You can use replicated volumes in environments where high-availability and high-reliability are critical.

http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Creating_Distributed_Striped_Volumes - Distributed striped volumes stripe data across two or more nodes in the cluster. For best results, you should use distributed striped volumes where the requirement is to scale storage and in high concurrency environments accessing very large files is critical.

http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Creating_Distributed_Replicated_Volumes - Distributes files across replicated bricks in the volume. You can use distributed replicated volumes in environments where the requirement is to scale storage and high-reliability is critical. Distributed replicated volumes also offer improved read performance in most environments.

Except when they do support redundancy:

http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Creating_Replicated_Volumes - Replicated volumes replicate files throughout the bricks in the volume. You can use replicated volumes in environments where high-availability and high-reliability are critical.

http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Creating_Distributed_Striped_Volumes - Distributed striped volumes stripe data across two or more nodes in the cluster. For best results, you should use distributed striped volumes where the requirement is to scale storage and in high concurrency environments accessing very large files is critical.

http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Creating_Distributed_Replicated_Volumes - Distributes files across replicated bricks in the volume. You can use distributed replicated volumes in environments where the requirement is to scale storage and high-reliability is critical. Distributed replicated volumes also offer improved read performance in most environments.

Use GlusterFS http://www.gluster.org/ for redundancy spanned across one or more JBOD machines for a much easier hardware and data upgrade path. Use oVirt for easy set up http://www.gluster.com/community/documentation/index.php/GlusterFS_oVirt_Setup_Guide. Mount GlusterFS directly to your clients or export via iSCSI target, fibrechannel target, FCoE, NBD, or traditional NFS for a more advanced shared storage solution. And you can still run more of a NAS type setup with CIFS, WebDAV, or the like.

Best part of acquisition: Gluster fsck

Unfortunately not it would seem according to this.

As your volume size grows beyond 32TBs, fsck (filesystem check) downtime becomes a huge problem. GlusterFS has no fsck. It heals itself transparently with very little impact on performance.

It was meant as a joke: Gluster rhymes with cluster, fsck is often used by Slashdotters for f*ck

Best part of acquisition: Gluster fsck

Unfortunately not it would seem according to this.

As your volume size grows beyond 32TBs, fsck (filesystem check) downtime becomes a huge problem. GlusterFS has no fsck. It heals itself transparently with very little impact on performance.

I've been on the market for a distributed, clustered file system for some time. Unfortunately, Hardoop is not really what I'm looking for. What I'm looking for:

1) Redundancy - no single point of failure.
2) Suitable for standard-sized file I/O.
3) Performance that doesn't completely suck ass.
4) Graceful re-integration when bringing a cluster portion back online.
5) Accessible through standard interfaces. (EG: Posix F/S)
6) Doesn't require a PHD in the technology to administer.
7) Doesn't require insane quantities of cash to build.
8) Stable.

There are clustered file systems that have some of these qualities. None that I've found so far have *all* of these qualities.

Hardoop fails on #1, #2, and #6. It has a single nameserver commanding the cluster, so if it goes down, well... (shrug) It also does poorly for "normal" sized files, somehow having a 10 GB file is the norm for Google. And setting a multiple node cluster up is definitely non-trivial.

Of all that I've reviewed, GlusterFS did the best but even in that case, I ran into severe over-serialization that brought my 6-node cluster to its knees. I tried three times to roll it out, and had to roll back all three times. I fiddled with the brick setup and caches for days before finally throwing in the towel.

Now I get by with rsyncing program files, and a homegrown data distribution setup using network sockets and xinetd. Not optimal to be sure, but so far it's scaled linearly and provides decent performance, at the price of a PHD in said technology. I guess you could compare our technology to MogileFS, only our scheme

A) uses DNS records to coordinate the cluster so that it scales up,
B) has a richer "where is the file" schema than the simple flat keys used by Mogile, and
C) has the ability to execute programs against files for performance. (EG: grep for searching text files, tar/gzip for compress/uncompress, virus scans, etc)
D) has the ability to "hang open" for activities like logging.

So far, this has held up well with about 500,000 file operations and millions of log entries per business day with an average file size of about 1-3 megabytes and every sign that growth can continue by simply stacking on more hardware. No, I'm not talking about massive throughput, but I *am* talking about the need for high availability systems that scale nicely without bottlenecks and exorbitant expense. Yes, it works pretty well, but we've had to invest significant programming time to do this.

Guess it's like the old engineering saw: Convenient, Cheap, Quality: pick any two!