Slashdot Mirror

http://wiki.lustre.org/index.php/Lustre_2.0_Features lists filesystem replication as a benefit of Luster 2.0 back in November 2009. I won't be running any RAID since my requirement isn't really to reduce number of disks used by relying on parity. One or more replication partners/mirrors will handle that function. Rsync won't work for the aforementioned clustered virtualization needs.

From their website:
http://wiki.lustre.org/index.php/Main_Page

High Performance and Scalability

For the world's largest and most complex computing environments, the Lustre file system redefines high performance, scaling to tens of thousands of nodes and petabytes of storage with groundbreaking I/O and metadata throughput.

Coda, and before that, AFS. Oh, and Lustre.

It's not a new idea. The only real difference here is that it's associated with BitTorrent and The Pirate Bay, and is designed to handle a whole set of problems you won't have, like untrusted machines communicating over the Internet, and how to compensate people for using their hard drive to store your stuff.

Oh my goodness, I thought that by now Lustre and ZFS had been much better integrated when Sun bought them. That is why I thought that with ZFS you could treat it like a CFS. I work for a federal lab and it was very hyped to us for about a year. I just googled and found this message:

http://lists.lustre.org/pipermail/lustre-discuss/2008-March/007006.html

Containing these nuggets:

---

Wed Mar 19 18:05:27 PDT 2008

> do you know if there is any chance that there be working lustre software for
> Solaris by the end of April?

Sorry, there is no chance of this at all. Due to integration issues
this is more likely to be available around the end of the year.

> > This is completely alpha code, you probably don't want to use it at
> > this time.

It refers to this wiki, not updated for a year now:

http://wiki.lustre.org/index.php?title=Lustre_OSS/MDS_with_ZFS_DMU

---

In fact all ZFS work is off of the roadmap in Lustre. I guess the promises made in 2007 did not pan out. It all seems even more likely dead now due to the Oracle purchase.

Oh my goodness, I thought that by now Lustre and ZFS had been much better integrated when Sun bought them. That is why I thought that with ZFS you could treat it like a CFS. I work for a federal lab and it was very hyped to us for about a year. I just googled and found this message:

http://lists.lustre.org/pipermail/lustre-discuss/2008-March/007006.html

Containing these nuggets:

---

Wed Mar 19 18:05:27 PDT 2008

> do you know if there is any chance that there be working lustre software for
> Solaris by the end of April?

Sorry, there is no chance of this at all. Due to integration issues
this is more likely to be available around the end of the year.

> > This is completely alpha code, you probably don't want to use it at
> > this time.

It refers to this wiki, not updated for a year now:

http://wiki.lustre.org/index.php?title=Lustre_OSS/MDS_with_ZFS_DMU

---

In fact all ZFS work is off of the roadmap in Lustre. I guess the promises made in 2007 did not pan out. It all seems even more likely dead now due to the Oracle purchase.

Or perhaps how you set a quota?

You can set a quota on a per filesystem basis. If you mean how to set a per user quota, you can't really do that yet but it's coming. There's nothing stopping you from creating a filesystem for each user and then assigning a quota to that filesystem.

Or perhaps how you do HSM?

How's this on ZFS and HSM?.

Or perhaps how you run it in a cluster environment?

If you're interested in high availability there are options with Sun Cluster (which is free) and ZFS. If you need a cluster file system that's a whole different beast. Might want to read this ZFS for Lustre information.

It looks like you're in the UK. Did they start censoring websites such as Google so you couldn't answer your own questions?

AFS is only about 20 years old, and supported on Windows, Mac, and most flavours of *NIX,

AFS as a standard may be 20 years old, but are there any actively-maintained, feature-complete, open source, and cross platform implementations around? I looked into using AFS about four years ago and decided that the answer to that question was "no".

OK, maybe it's time to give it another chance...

The Wikipedia entry for AFS says:

There are three major implementations, Transarc (IBM), OpenAFS and Arla, although the Transarc software is losing support and is deprecated. AFS (version two) is also the predecessor of the Coda file system.

Transarc is off the list straight away, then. On to OpenAFS, the open-source fork of Transarc. Seems to be actively maintained, the last development release being within the last few weeks... but:

1.5.53 is also the most recent in the series of releases intended to provide new experimental features including the Demand Attach File Service and Disconnected AFS

Disconnected operation is still "experimental". Not encouraging.

On to Arla. Seems to be a client-side only implementation and hasn't reached a 1.0 release yet. Next!

So, AFS 2 became Coda? The Wikipedia entry for Coda states that Coda has been superceded by InterMezzo, another dead project that was dropped from the Linux 2.6 kernel and has itself been superceded by Lustre... which actually looks damn impressive, but is designed for Linux HA clusters, and doesn't have the Windows and Mac support that AFS does.

So yeah, still can't find an actively-maintained, feature-complete, open source, and cross platform distributed filesystem. Would love to be proven wrong though; any takers?

Sun is about to lose the enterprise completely and they don't even know it. [...] Lustre--which only works on Linux--is what is really needed to take on this decade's problems.

-- parent post

Lustre is a scalable, secure, robust, highly-available cluster file system. It is designed, developed and maintained by Sun Microsystems, Inc.

-- http://www.lustre.org/

Hmm...

OK, we got a half way overview of CERN's decision, with some bold statements of questionable validity. I am submitting the criticism purely on the grounds of being really interested in large data storage, I don't work for any large storage vendor, but I am an architect of storage systems.

First of all, with the statement "and it's (StorNext) completely vendor independent": Lot's of other solutions provide flexibility about choosing the hardware vendor from a theoretical perspective. The theory says that if vendor A makes a SAN, vendor B makes a RAID controller, C a disk cabinet and D offers a clustered FS, and all comply to the relevant standards, you can plug them together and expect them to function. However, imperfections in the standards, hidden proprietary optimizations, always dictate certain configs and combinations for optimum performance. There is a lot of work to be done in the StorNext and other similar products, until they claim full flexibility. My experience in deploying a StorNext based solution on a 1200 node setup says so and to keep the post short, I shall exclude at this stage vendor details, but if someone is interested, I am happy to go over the details. There is vendor dependence if you wish optimum performance. Not to mention that if you mix and match the RAID and SAN cards in the setup, any unfortunate issue might end up in a multi-headache, even if you have solution support (A blaims B, B accusses A, and the game of ping-pong begins). You can never exclude vendor dependence in such a large setup, you have to deal with it.

Then you have the "Clustered file systems are still an evolving category, she says, but enterprise IT is warming up to it.". I can imagine what the author classes as enterprise IT here, but I think there is a bit of an orientation issue. CERN is not exactly the classical enterprise IT environment, is it? Not in terms of their requirements for resilience and capacity. These FAR EXCEED enteprise IT requirements. CERN is a research setup. And the mentality of a research setup (that incubated the WWW after all) is (or should be) that of innovation and playing with some of the latest and the greatest. In fact, some US based research setups have long experimented with other cluster FSes. They are not warming up. CIO claims that StorNext is scalable. It is. But to what extent? Have they excluded for example things such as Lustre? http://wiki.lustre.org/index.php?title=Main_Page If yes, why?

I am really surprised they did not use the Lustre filesystem for their data storage since it is vendor neutral, open, and designed for exactly this sort of thing. The lustre guys report being able to obtain tremendous bandwidth and scalability. I have not yet been able to play with Lustre but I look forward to doing so.

Any of the folks in the loop know if they use Lustre for their storage backend?

Unfortunately, it's not even close. A couple things that spring to mind:

AFS servers don't automatically ensure that every piece of data is stored on some minimal number of them. Enabling/configuring replication of a given volume is a manual process.

AFS treats a single server as the "master" read-write copy; replicated copies are read-only.

Redhat's Global File System ( http://www.in.redhat.com/software/rha/gfs/index.ph p3 ) or Lustre ( http://www.lustre.org/ ) or maybe ddraid ( http://sources.redhat.com/cluster/ddraid/ ) sound much closer to what googlefs is than AFS does.

IMO AFS is more of a file sharing protocol (like NFS or SMB) than googlefs.

It's been a while since I built a mid-size email system, but the last time I did it I used:

Data stores were maildirs on NetApps
SMTP servers running Postfix
IMAP servers running Courier IMAP
Logins via NIS
IMAP and SMTP failover by means of load balancers

The SMTP and IMAP servers get NIS-distributed automounter tables, so everyone's homedir is available everywhere. The load balancers distribute the load out to the SMTP and IMAP servers, and work around any that fail. Mail comes into the SMTP servers, and Postfix delivers to maildirs in the users' homedirs. Any SMTP server can deliver to any user. Users log in with IMAP on the Courier IMAP servers. Again, all homedirs are everywhere, so it doesn't matter which server they hit.

Adding capacity at any point is easy - you just add more servers of the appropriate type when you need more. IMAP and SMTP are fully redundant. Load balancers usually only operate in failover pairs, but you can add more A records in DNS for more LB pairs if you need it.

The one sticky point is the data stores on the NFS servers. Adding capacity is easy (just add more servers). but there's no easy way to make this fully redundant. See notes for more.

So there you have it. That'll scale to a pretty large system, and it's simple to implement. It's not THE MOST scalable system, but if you have to ask, this is probably sufficient for your needs.

Notes:

You must use maildirs, not mbox. Maildirs perform very well even on NFS, because there can be multiple simultaneous readers and writers. mbox requires locking.

With NetApp, or Red Hat Cluster Server, or any other cluster NFS server, you can make the head end redundant, so your disk shelf becomes the last single point of failure. If you run RAID 1+0, you can have all the disks mirrored across two shelves, so at least the hardware is completely redundant. However, there are still rare, but possible failure modes. STONITH is, ultimately, a problem that has no perfect solution. (Look it up if you're not familiar with STONITH.)

NetApp makes very reliable NFS servers. Even in single head configurations my uptime experience has been incredibly good. Dual head is even better. But they're god awful expensive. There are other ones you can buy at all different price points. Clustered file systems like Coda sound really sexy, but they're still half baked. Lustre http://www.lustre.org/ might work well, but it wasn't available when I last did this, so I can't say. Choose what's appropriate to your needs and budget.

I used NIS. These days LDAP is more fashionable. Make your LDAP server redundant of course.

You need redundant networks. In the simplest case, put half of each type of servers (IMAP, SMTP, LB, NFS) on two different switches.

I never bothered with POP, but you can get POP servers for maildirs, too.

Configure your load balancers to balance per session - IE, if a user creates multiple IMAP connections, they all go to the same server. This helps keep down the number of NFS mounts, LDAP requests, etc.

Software opinions: I like Postfix and Courier. They're simple, robust, flexible enough for most situations, and perform very well. Cyrus also has a good following in the large-scale arena, but does things different. Qmail's non-OSS license prevents people from releasing versions that strip out djb's quirky way of doing things, which is why I left it for Postfix (and never looked back). Sendmail doesn't suck as much as it used to, but I haven't really seen why I SHOULD use it these days either. Any of these can be made to work, though, so use whatever you're comfortable with.

Tip for any email system: outright reject (IE, don't accept at all, don't send to someone's spam folder) as much spam as you can. If 90% of your mail is spam, and you reject the 90% most-likely-spam (delivering the other 10% more questionable stuff to a spam folder), you've just increased your mail performance and disk space by > 5x.

Good luck!

It would be nice to see comparisons to RedHat/Sistina's GFS, Lustre (backed by HP), and others listed here.

Also how does this compare to clustered storage that is not run on the hosts themselves like NetApp new Spinnaker based clustering. You also have folks like Isilon, Panasas, and Terrascale.

Anybody have an good data on this?
-Ack

You must have missed the part where I said, "High performance data archive."

Stuff I am talking about costs 1000 times more.

No it doesn't: http://www.lustre.org/

Clustered filesystems are your friend. They give you speed and flexibility unattainable by other filesystems.

Further, why does Apple need "ultra expensive super fast storage"? I think you're getting a little too excited about the idea of serving that many songs and forgetting the basic logistics. They sell, what, a million songs a month? At ~4MB a piece, they're pulling 4T a month off disk. Yes, I know, they serve previews for every song, etc, but let's just tackle these issues one at a time.

4T a month is 133 megs a day. Hell, you COULD string 100 Seagate drives together and pull 133 megs per SECOND off the array.

So how many previews are they streaming? I bet that's where the mass of their I/O is. If they serve 10 previews for every song that's 10 million per month, or 3.8 per second. They can't be more than half a meg each. So 1.9 megs per second of I/O. Of course this isn't linear, the data comes and goes in bursts, but tell me again how their arrays have to be "EXTREAMELY high I/O" and "1000 times more" expensive?

This is a simple solution. You don't need to spend tens of millions of dollars on storage and, quite frankly, I doubt Apple did. You're challenging me on this issue but I'm not sure you realize that high performance, highly available, very large storage is my job. It's what I do, every day. I analyze storage needs, recommend and implement solutions. And I don't waste my employer's money on expensive fiber channel SANs when there's no chance they'll ever need the bandwidth.

The filesystems going to be the hardest component of this. I know of no open-source fs that could handle this.

Be enlightened: http://www.lustre.org/

"The latest version of Lustre is always available from Cluster File Systems, Inc. Public Open Source releases of Lustre are made under the GNU General Public License. These releases are found here, and are suitable for clusters with thousands of nodes and hundreds of terabytes of storage."

GPFS
Take it from someone who's messed with nearly every storage product on the market, if you want something that works fairly simply, performs at approaching spindle speed ( meaning the file system is not the bottleneck - if you have 10 GB/sec. storage bandwidth, expect to see near that with proper tuning ), is very stable ( compared to most storage solutions on the market - bear in mind that most storage products are aimed at large-block sequential I/O, and fall down - either performance-wise or stability-wise - when you throw other I/O patterns or combinations of patterns at them ), and is portable across nearly any Linux distribution ( with varying amounts of difficulty, I have had to hack their kernel patches before when using a unsupported kernel ), GPFS is the one. Of course, the problem there is I believe it's pretty expensive to run on non-IBM hardware. But if you have IBM hardware ( even if it's not the hardware you're running the FS on ) or some sort of in with IBM, they'll let you have it for a song and a dance.

Having said that, Lustre is getting there. I'd say it's the equal of GPFS ( as a parallel filesystem - I believe it is even more flexible as a distributed filesystem ) in performance, probably scales roughly the same ( haven't played with it in a large installation, so can't tell you beyond looking at the architecture ), and is going to the be the biggest player on the market in the future. It's also free ( IIRC Cluster File Systems sells support, but the code is freely available ) and not tied to IBM and whatnot, like GPFS is. Of course, HP has a big connection with Lustre, but not ownership thereof.

Those are really the only two that I would consider for a serious high-performance storage project. If you don't need great performance, that's when you can start looking at things like GFS, ADIC's StorNext, Ibrix, etc.

Oh, Gautham Sastri ( of former Maximum Throughput fame ) has a newer company called Terrascale, I recall them putting on a presentation at the 2003 or 2004 ( can't remember ) Supercomputing conference ( SC2005 is coming up in a few weeks, yeah!!! ) which showed pretty good performance ( relative to the small system they were using ), not sure how they're coming along...

Anyways, good luck...and don't forget to use Iozone to benchmark the damn thing!

IDE and SATA are not a choice for anything but a low end low speed/load server.

BS.

http://www.lustre.org/

Check out Lustre at http://www.lustre.org/ It's being developed/used by the DOE on alot of Supercomputer Cluster systems, for multi-terabyte storage stuff.

Right now there are a lot of file systems that do somehing not all that different than what Sun is proposing. The project I am on is evaluating them as we speak for a center wide filesystem. I've had the fun (no sarcasm, honestly) of setting up a number of different onces and helping to run benchmarks and tests against each. All of them have strengths. Every single one of them has some nasty weaknesses.

If you are looking for an open source based cluster file system, Lustre is what you want. It's supported by LLNL, PNNL, and the main writers at ClusterFS Inc. It's a network based cluster FS. We've been using it over GigE. However, we've found that there needs to be a ratio of 3:1 for data server:clients for a ratio. Wehave only used one metadata server. Failover isn't the greatest. Quotas don't exist. it also makes kernel mods (some good and bad) to do a mild fork of the linux kernel (they put them into the newer kernels every so often). It only runs on Linux. Getting it to run on anything else looks...scary.

GPFS runs on AIX and Linux. Even sharing the same storage. It runs and is pretty stable. it has the option to run in a SAN mode or network based FS. In the latter form, it even does local discovery of disks via labels so that if a client can see the disks locally it will read and write to them via FC rather than to the server. It, however, is a balkanized mess. It requires a lot more work to bring up and run: there is an awful lot of software to configure to get it to run (re: RSCT. If you haven't had the joys of HATS and HAGS, count yourself very, very lucky).

ADIC's StorNext software is another option. This one is good if you are interested in ease of installation, maintanence, and very, very fast speeds (damn near line speed on Fibre channel). I have set this one up for sharing disks in less than two hours from first install to getting numerous assorted nodes of different OS's to play together (Solaris, AIX, Linux). It freakin on virtually everything from Crays to Linux to Windows. It's issues seem to be scaling (right now doesn't go past 256 clients) and it has some nontrivial locking issues (righting to the same block from multiple clients, and parallel I/O to the same file from multiple clients if you change the file size).

There are some others that are not as mature. Among them are Ibrix, Panasas, GFS, and IBM's SANFS. All of them are interesting or promising. Only SANF looks like it runs on more than Linux though at this point. Our requirements for the project I am on are to share the same FS and storage instance among disparate client OSes simultaneously. This might not be the same for others though and these might be worth a look. Lustre dodges this because its open source and they're interested in porting.

Or Lustre?

Dunno how I missed it in preview... http://www.lustre.org

Lustre, I don't know anything about it except it exists. SMB, yes it's the spawn of satan, but if you need to share with windows machines there's really no choice.

GFS was well-liked at supercomputing centers I have worked with until Sistina dropped the GPL license in favor of proprietary. They did this very suddenly and without warning. It pissed off a lot of potential users and the open source community. It has since fallen out of favor.

This move by Red Hat gives new life (and resources) to GFS beyond the OpenGFS Project that has also been continuing to work on the code.

Another recent development in this area is HP's decision to productize Lustre. Lustre is perhaps the most prominent and promising HPC filesystem.

SGI also announced a major deal last week involving Luster:

The new file system is expected to sustain write rates in excess of 8GB/sec and demonstrate single client write rates of more than 600MB/sec. To achieve this performance, the new file system will leverage Lustre, an open source, object-oriented file system with development lead by Cluster File System Inc., with funding from DOE. Lustre currently is used on four of the top five supercomputers, including the PNNL cluster based on 1,900 Intel® Itanium® 2 processors.

If you could rebuild everything from the ground up (and had tons of money to throw at it), you'd most likely want to build a system based on a very expensive vendor solution.

Assuming that you can't do that, your best bet would be to go with some sort of parallel filesystem, the likes of Lustre, GFS, Ibrix, GPFS or CxFS. The architectures of these vary, but the basic principle they share is performance scalability based on increasing the number of data paths to the disk. So if you have, say, 100 nodes on a high-speed network, you take 10 of them and attach them to your SAN. The parallel filesystem spans the entire SAN and so requests from the nodes can reach any bit on the SAN from any of the ten paths. If you need more performance, you add more paths: controllers, HBAs and storage nodes. I know GPFS scales linearly in performance based on the number of paths to the data, and I believe the others scale well also.

I haven't hit 50 TB on disk (I have on tape, but your post suggests that tape wouldn't give you the performance you need), but I have set up several 4-8 TB GPFS filesystems that could easily grow to 50 TB if I had the spindles.

Good luck finding a solution; symlink-based solutions on a convnentional *NIX filesystem are a nightmare; I sympathize.

I guess that's what they are doing with Lustre.

Only then Linux is de 'file-server'.

They are 'pretty' much done, if you pay Cluster File Systems Inc. (US$ 5000), they'll give you the latest version and support if something doesn't work.

You could say, they are between stable and beta-testing (lot's of things are done, just a number of corner cases that there current customer-base doesn't need yet).

Lustre is something we're looking at rolling out for user home directories. Although a few labs have 100TB+ file systems using it. You get redundant servers at all levels (which deals with the synchronization problems), and best of all, you can stripe all your existing disks to create one logical disk. Think LVM for network connected machines. It's pretty fast too.

You'll be wanting a distributed cluster filesystem. There are several available, with their pros and cons. They are also all aimed at enterprise / HPTC installations. For home use you'll be better off with a set of RAID disks.


GPFS from IBM. This is free for academic use, but you pay for commercial use. Linux or AIX only.


GFS from sistina. Commercial offering. Linux only.


Lustre. This is beta quality code, but is freely available. It might work wonderfully, or it might eat your files.


(open)AFS. Works, but has limitations. It does not support large files and clients aren't available for all OSes.

The lustre project (www.lustre.org) is supposedly going to be the end all/be all of distributed parallel file systems, but I believe it is still fairly unstable and not ready for production use. In the meanwhile, the best one out there is PVFS(www.parl.clemson.edu/pvfs/). Fat chance trying to find Windows clients, but you can always re-export it with Samba.

The highest performance is probably from Lustre, although it is designed for slightly larger clusters. Haven't tried it yet though.

What is it using for disk space? MCR @ LLNL is using the Lustre file system with DataDirect Networks storage.

http://www.lustre.org

that "10-20" terabytes line has to be a typo.
I spoke w/ some people from CERN regarding their CASTOR HSM, and a few years ago they were up in the petabyte range already. By now, they're probably sitting at at least a few hundred TB online, and probably 5 PB offline, as a conservative guess.
IBM's been doing GPFS filesystems in the > 50 TB size, w/ > 1 GB/sec. throughput for years. That, and even's IBM's mid-tier FAStT products can confortably carry 12 TB on one dual-controller storage head.
Still, further abstracting the issue of locality is very exciting stuff. I'd be interested to see exactly how they go about doing it, and if it's anything that you can't get w/ Lustre when it's ready.

Lustre which grew out of the Intermezzo/Coda projects. It was designed from the ground up as a scaleable ditributed filesystem. I would give a single mount point for a set of aggregated storage. In our clustered system it has been show to be able to write greater than 2 GB/s to disk. Lustre may be a little overkill for a home system, but for clusters, it works great.

I believe that Peter Braam (one of the authors of InterMezzo) has been working on a new file system named Lustre that builds on top of it's ideas. It's out in beta There is a detailed document here. Braam gave a nice talk on it at LinuxWorld this last January. I haven't tried it yet.

I believe that Peter Braam (one of the authors of InterMezzo) has been working on a new file system named Lustre that builds on top of it's ideas. It's out in beta There is a detailed document here. Braam gave a nice talk on it at LinuxWorld this last January. I haven't tried it yet.

For more information, oddly 'nuff, you can visit www.lustre.org. :)

Actually, they have a specially written parallel FTP client/server that stripes large pieces of data across multiple physical connections. Remember, we're talking about data sets that can be terabytes in size. To do this type of thing more efficiently, they've put together lustre. Check it out. It's no NFS/CIFS. It's supposedly the next step after IBM's GPFS.

If your home computer is online 24/7 (which is presumably is if you're on broadband) t's cooler to use SAMBA, AFS (or here), Coda, InterMezzo, NFS, or the unfinished Lustre. If you're not big on effort, set up an http or ftp (or gopher!) server. That way, you have an automatically up-to-date menu of your mp3s, where you can access all your music any time you can connect to the 'Net.
This box is just itching to be a Coda server.

What we have done is split the ext2 filesystem into two layers:

• the VFS interface, which knows about files and directories
• the disk interface, which knows about files, blocks, and bitmaps

What this allows is for the on-disk data handling to be abstracted from what the mounted filesystem looks like, so we can do RAID/mirroring, snapshots, RPC (remote access like NFS), encryption, etc, just by stacking a small mid-level driver between the filesystem and the disk. Currently we have snapshots implemented, and RPC is in progress. The snapshots let you mount several "versions" of your filesystem, with all of the older copies being read-only filesystems.

It is definitely NOT for production yet, but OK to play with. A regular OBDFS filesystem can be mounted as ext2 and vice versa, but a snapshot filesystem cannot be mounted/fscked by the normal ext2 tools, unfortunately, although this may change in the future.