Slashdot Mirror
Distributed Storage Systems for Linux?
elambrecht asks: "We've got a _lot_ of data we'd like to archive and make sure it is accessible via the web 24/7. We've been using a NetApp for this, but that solution is just waaaay to expensive to scale. We want to move to using a cluster of Linux boxes that redundantly store and serve up the data. What are the best packages out there for this? GFS? MogileFS?"
Check out Lustre at http://www.lustre.org/ It's being developed/used by the DOE on alot of Supercomputer Cluster systems, for multi-terabyte storage stuff.
Panasas http://www.panasas.com/products_overview.html has some products which probably fit your requirement of high speed distributed storage.
- What's "a lot"? 1MB is a lot of data if you think about it. When people start talking about "a lot" of data these days, I assume they're meaning hundreds of terabytes. Is that what you mean?
- What's the budget? What performance do you need? Do you need to back it up? Do you need to replicate it? Your post is sort of like "hi, I have a problem. What is the answer? Thanks!"
Also, it's "too expensive to scale," my friend. You'd think an "Editor" like Cliffy would fix posts, but he's too lazy.If you can afford NetApp, why not keep with NetApp? A bunch of Linux boxes is not a storage solution. Indeed, what does Linux have to do with anything? We're talking storage here. What are you planning to do - put in 200 of them with internal SATA drives? Yeah, that'll be a lot cheaper to maintain...
I'm not shilling for NetApp, but if you really have "a lot" of data to put "on the web" "24/7" then you need some kind of real storage solution like a NetApp or one of their competitors.
Now go away and please take Cliff with you.
Advice: on VPS providers
How about OpenAFS ? It is sort of like NFS on steroids, with redundancy, scaling, cacheing, Kerberos-based security ... I've just started looking at it myself, but it seems pretty slick.
10b||~10b -- aah, what a question!
NetApp is number four in storage revenue terms, after EMC, HP and IBM
so go ask them about what you want
really you can admin your white box's (that become a NAS ) or you can get a NAS
are you thinking SAN ?
also talk to Apple they do some nice product as well as SUN
whats this for large data ?
video data go talk to SGI and their XFS products
really it depends on what your doing NetApp is great for company File system of documents but Bad if you want to get the most out of your storeage and you do mostly video/music dont care about snapshots etc....
regards
John Jones
Get A Centera.
I'm biased but this is a high level Linux based storage system done right. It's not easy to create a coherent storage system out of lots of separate machines, the software that runs on this cluster does a lot of work. This thing fully redundant with no single point of failure, dynamically expandable without even taking it offline, it scales to 100's of terabytes and manages all that content continuously (scanning for corruption and fixing it, garbage collecting, etc..). The cluster has redundant backend networks and parallel paths everywhere, it even uses reiserfs to store the data. There's a lot of good engineering in this unit and they sell it at a decent price compared to NAS boxes.
Check it out:
http://www.emc.com/products/systems/centera.jsp
I do work for EMC (like I said.. I'm biased) but I don't speak for them, my opinions are my own.
Storage clustering is simply hard to do while still presenting a low level filesystem interface. Tossing that out and creating file storage as a high level service with a richer interface seems like the right approach to me. Show me a storage clustering solution that doesn't do that and I'll show you something full of bugs, expandability issues, limitations, and pain points.
set softtabstop=4 shiftwidth=4 expandtab nocp worlddomination
I'm sure they'd be happy to sell you something along the line of serving data....
I did some reasearch on clustering filesystems for work a while ago. Here's the Cliffs-notes version:
GFS High-end, a pain in the ass to set up and run. Wants a RHEL server or two to run. OpenGFS Started as a fork of GFS when the GFS license changed, it has followed a bit of a different path. Not nearly as stable or fast as GFS, but might be there some day. Lustre Lustre should be really nice, but is horrendous to run (at least, that's the word from my friends at Sandia, who know a thing or two about it). General consensus is that you need a full-time staff member just to make it work. If you can afford that, it's a good way to go. PVFS Fast, light-weight, not POSIX-compatible. If your apps don't need the stuff it doesn't do, or you're willing to write some glue code for your app to speak PVFS natively instead of using the FS driver, this is a great way to go. Looks simple to set up (as simple as these things get).^]:wq
A barely-related subject - I've been wondering whether there's some way to collect the unused space on all the Windows workstations around here into a shared space for storage.
This is purely a speculative exercise, but I keep wondering if some combination of:
Yes, I know it's kind of silly, and performance seems like it would be pretty pathetic, but the more I think about it, the more I want to see if I could actually do it (think pretty much the same mindset that the IP-over-carrier-pigeon guys had...)
Heck, it might conceivably actually WORK for a large-but-infrequently-accessed historical repository or something...
Or has someone already started some sort of "Virtual ATA-over-ethernet-from-a-file driver for Windows" project and spoiled my fun?...
Hacker Public Radio is our Friend
We have about 27TB of data from Mars (and adding another TB per month) that we need to keep online. We have been using netapps, but at ~$25K/TB, plus maintenance (3 years maintenance is about as much as a whole new system) they're just WAY too expensive for data warehousing.
We've moved to using linux based OpenAFS servers. A high quality 3U box (qsol.com) loaded with 16x 300GB ATA drives costs about $8.5K and provides us about 3.5TB (2 drives for parity, 2 drives for hot-swap). That works out to $2.5K/TB. If your risk tolerance is higher than mine, you can bring that up to $8K/5.5TB, for about $1.5K/TB). We really want 99.999% availability, so just to be safe, we keep a 100% redundent read-only copy on a second machine (AFS supports this beautifully, including automatic fail-over).
OpenAFS has a couple of features that make it better than NFS (client-side cache, for instance), but it also has a few drawbacks, like no files >2GB.
Check out http://www.ibrix.com/ This is a perfect solution for your requirements. Pixar uses this.
Every day is Saturday and all the rainbows have silver linings.
This is the solution archive.org uses.
http://www.archive.org/web/petabox.php
They are on the order of petabytes
-- I was raised on the command line, bitch