Distributed Storage Systems for Linux?

elambrecht asks: "We've got a _lot_ of data we'd like to archive and make sure it is accessible via the web 24/7. We've been using a NetApp for this, but that solution is just waaaay to expensive to scale. We want to move to using a cluster of Linux boxes that redundantly store and serve up the data. What are the best packages out there for this? GFS? MogileFS?"

Comments

AFS ??

[forsetti]

How about OpenAFS ? It is sort of like NFS on steroids, with redundancy, scaling, cacheing, Kerberos-based security ... I've just started looking at it myself, but it seems pretty slick.

Ask Google

[VernonNemitz]

I'm sure they'd be happy to sell you something along the line of serving data....

Converting extra Windows(tm) workstation space?

[Dr.Dubious DDQ]

A barely-related subject - I've been wondering whether there's some way to collect the unused space on all the Windows workstations around here into a shared space for storage.

This is purely a speculative exercise, but I keep wondering if some combination of:

• Every Windows(tm) workstation "shares" an otherwise-empty subdirectory
• a Linux box creates and uses a "filesystem image" file of some kind ("loopback mount"-style image) stored on each share over SMB/CIFS
• Linux uses VFS to combine the individual virtual drives into a larger drive (or perhaps two identical-size virtual drives, which are then combined into a single software RAID 1 array?)
• Linux then shares this Rube-Goldbergian system as a Samba share...

Yes, I know it's kind of silly, and performance seems like it would be pretty pathetic, but the more I think about it, the more I want to see if I could actually do it (think pretty much the same mindset that the IP-over-carrier-pigeon guys had...)

Heck, it might conceivably actually WORK for a large-but-infrequently-accessed historical repository or something...

Or has someone already started some sort of "Virtual ATA-over-ethernet-from-a-file driver for Windows" project and spoiled my fun?...

Re:Converting extra Windows(tm) workstation space?

[egarland]

If you want to try building it, I'd suggest you start with a nice high level method of creating linux based filesytems:

http://perlfs.sourceforge.net/

Build it first, optimize later.

FYI.. The multi-threaded filesystem version exists, I just haven't bundled it up pretty for distribution. Now someone needs to create a multi-threaded samba to share it out.