Distributed Storage Systems for Linux?

elambrecht asks: "We've got a _lot_ of data we'd like to archive and make sure it is accessible via the web 24/7. We've been using a NetApp for this, but that solution is just waaaay to expensive to scale. We want to move to using a cluster of Linux boxes that redundantly store and serve up the data. What are the best packages out there for this? GFS? MogileFS?"

Comments

Lustre

[Yerase]

Check out Lustre at http://www.lustre.org/ It's being developed/used by the DOE on alot of Supercomputer Cluster systems, for multi-terabyte storage stuff.

Panasas -- check it out

[middlemen]

Panasas http://www.panasas.com/products_overview.html has some products which probably fit your requirement of high speed distributed storage.

look at the rest NetApp is 4th place

[johnjones]

NetApp is number four in storage revenue terms, after EMC, HP and IBM

so go ask them about what you want

really you can admin your white box's (that become a NAS ) or you can get a NAS

are you thinking SAN ?

also talk to Apple they do some nice product as well as SUN

whats this for large data ?
video data go talk to SGI and their XFS products

really it depends on what your doing NetApp is great for company File system of documents but Bad if you want to get the most out of your storeage and you do mostly video/music dont care about snapshots etc....

regards

John Jones

Centera

[egarland]

Get A Centera.

I'm biased but this is a high level Linux based storage system done right. It's not easy to create a coherent storage system out of lots of separate machines, the software that runs on this cluster does a lot of work. This thing fully redundant with no single point of failure, dynamically expandable without even taking it offline, it scales to 100's of terabytes and manages all that content continuously (scanning for corruption and fixing it, garbage collecting, etc..). The cluster has redundant backend networks and parallel paths everywhere, it even uses reiserfs to store the data. There's a lot of good engineering in this unit and they sell it at a decent price compared to NAS boxes.

Check it out:
http://www.emc.com/products/systems/centera.jsp
I do work for EMC (like I said.. I'm biased) but I don't speak for them, my opinions are my own.

Storage clustering is simply hard to do while still presenting a low level filesystem interface. Tossing that out and creating file storage as a high level service with a richer interface seems like the right approach to me. Show me a storage clustering solution that doesn't do that and I'll show you something full of bugs, expandability issues, limitations, and pain points.

Clustering filesystems- an overview

[houdini_cs]

I did some reasearch on clustering filesystems for work a while ago. Here's the Cliffs-notes version:

GFS High-end, a pain in the ass to set up and run. Wants a RHEL server or two to run. OpenGFS Started as a fork of GFS when the GFS license changed, it has followed a bit of a different path. Not nearly as stable or fast as GFS, but might be there some day. Lustre Lustre should be really nice, but is horrendous to run (at least, that's the word from my friends at Sandia, who know a thing or two about it). General consensus is that you need a full-time staff member just to make it work. If you can afford that, it's a good way to go. PVFS Fast, light-weight, not POSIX-compatible. If your apps don't need the stuff it doesn't do, or you're willing to write some glue code for your app to speak PVFS natively instead of using the FS driver, this is a great way to go. Looks simple to set up (as simple as these things get).

The IBRIX file system is a strong runner for this.

[schnook]

Check out http://www.ibrix.com/ This is a perfect solution for your requirements. Pixar uses this.

Re:We use OpenAFS

[luizd]

Not anymore in OpenAFS 1.3.81.

Copied from release notes:

For UNIX, 1.3.81 is the latest version in the 1.4 release cycle. Starting
in 1.3.70, platforms with pthreads support provide a volserver which like
the fileserver and butc backup system uses pthreads. Solaris versions 8
and above, AIX, IRIX, OpenBSD, Darwin, MacOS and Linux clients support
large (>2gb) files, and provided fileservers have this option enabled.
HP-UX may also support large files, but has not yet been verified. We hope
sites which can do so will make use of 1.3.81 on their UNIX platforms and
provide feedback to help us fix any remaining issues before 1.4 is
released.

aRchive.org

[fulldecent]

This is the solution archive.org uses.

http://www.archive.org/web/petabox.php

They are on the order of petabytes

Re:Converting extra Windows(tm) workstation space?

[LuckyStarr]

[...], but I haven't done any digging to find out if it's possible to directly create a ('standard') filesystem as an image file. (Hints welcome...)

Huh? Just run mkfs.whatever on your file. Should work without problems. Your filesystem is as large as it would be on an equally large blockdevice.

Example:

$ mkfs.ext3 file
mke2fs 1.36 (05-Feb-2005)
file is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
1784 inodes, 7116 blocks
355 blocks (4.99%) reserved for the super user
First data block=1
Maximum filesystem blocks=7340032
1 block group
8192 blocks per group, 8192 fragments per group
1784 inodes per group

Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 38 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.