Slashdot Mirror

← Back to Stories (view on slashdot.org)

PVFS2 - a High-Performance Parallel File System

Posted by timothy on from the good-nodes-are-still-available dept.

neillm78 writes "As part of the development team, we're announcing PVFS2 version 1.0 here in Pittsburgh at the SC2004 conference! PVFS2 is a GPL/LGPL based parallel file system for cluster-based applications. It logically groups any number of storage servers into a coherent file system for use by client nodes, specifically tailored to handle efficient access to large shared files. PVFS2 supports access via an MPI-IO interface for high-performance parallel applications, but you can still mount it like a regular GNU/Linux file system for traditional serial applications and managment. The PVFS2 project is conducted jointly between The Parallel Architecture Research Laboratory at Clemson University and The Mathematics and Computer Science Division at Argonne National Laboratory. Please feel free to give it a try!"

10 of 26 comments (clear)

  1. Been following it for a while... by brsmith4 · · Score: 2, Informative

    PVFS (in its first incarnation) despite some instability (more so due to the fact that our first cluster was COTS cheap-o hardware), really helped drive down the load on our clusters by removing the need to perform NFS writes to a single head node for scratch space. The set up is extrememly simple and the code base was really small.

    I plan on evaluating PVFS2 for our new clusters along with Lustre and GFS although I have heard nothing about the latter two operating over the MPI-ROMIO subsystem (which would definitely offer a performance increase).

    1. Re:Been following it for a while... by brsmith4 · · Score: 4, Informative

      It's a parallel file system, not a drop in replacement for local FS's like XFS or ext3. It runs across multiple hosts, striping the data on each host. Also, haveing multiple I/O hosts in the array helps to distribute the read/write across multiple nodes, thus reducing the overhead for those operations.

      This is like "Distributed NFS" although that description does it a huge injustice, it should help to get the point across.

    2. Re:Been following it for a while... by mikefe · · Score: 2, Interesting

      So it works over the network without needing a network block device layer?

      That would mean it should compete on the level of OpenAFS, Intermezzo and CODA for fault tolerant network filesystems -- except it would have internode locking which the others don't at the moment.

      That would also mean it doesn't directly compete at the same level as GFS (which is targeted at configurations of servers connected by a SAN or similar).

      Is this project set on integrating with the mainline kernel? What has/will happen on that front?

      This also looks perfect for an active/active LinuxHA failover cluster -- if it has redundancy, which any clustering filesystem should have. Right now the LinuxHA project is integrating GFS into their stack of interwoven sub-projects.

      After looking at the site, it looks like it would be good for server to server connections, and not good for server to workstation connections. For instance, it doesn't look like it has any caching functionaility like OpenAFS does and it looks like each node needs to have a copy of some of the cluster data (or does that end at the meta-data nodes?). PVFS2 looks like it has a similar archatecture to Lustre, except PVFS2 is developed openly.

      --
      There: Something at a specific location.
      Their: Owned by someone.
      Please make sure your english compiles.
    3. Re:Been following it for a while... by rizzy · · Score: 2, Interesting

      That would mean it should compete on the level of OpenAFS, Intermezzo and CODA for fault tolerant network filesystems -- except it would have internode locking which the others don't at the moment.

      That's an interesting thought, but at no time have we ever thought of ourselves as a replacement for those file systems. The ones you mention are general purpose file systems whereas PVFS2 is meant to be a fast file system for parallel applications.

      except it would have internode locking which the others don't at the moment.

      I'm not sure what you mean here. We have no locking anywhere -- which is exactly why we can deliver such high performance. Scientific applicaitons often don't need a locking subsystem getting in their way.

      Is this project set on integrating with the mainline kernel? What has/will happen on that front?

      There really isn't much for us *to* integerate into the kernel. We do have a VFS interface, but it acts primarily as a way to convert kernel system calls into userspace PVFS2 calls. Yes, there are lots of "file system in userspace" projects, but by making something that works just for PVFS2, we can get better performance.

      This also looks perfect for an active/active LinuxHA failover cluster -- if it has redundancy, which any clustering filesystem should have. Right now the LinuxHA project is integrating GFS into their stack of interwoven sub-projects.

      Funny you should mention LinuxHA. I spent some time this summer setting it up with PVFS2. If you really care about redundancy, you can invest in shared storage solutions (SCSI and firewire drives can be shared between two hosts simulaneously -- if you buy the really expensive stuff). With shared storage, you've got a way to tolerate node failure. You're still screwed if something eats your big expensive hard drive, granted. We're working on software replication.

      PVFS2 looks like it has a similar archatecture to Lustre, except PVFS2 is developed openly.

      Thanks for noticing! While I understand why CFS has taken the approach they have, we really feel that the HPC community (and Linux in general) needs a file system that's free software.

  2. It's Linux! by egarland · · Score: 2, Insightful

    The kernel is called Linux. Yea, you may compile against GCC but come on people! it's a Linux specific kernel module. Leave the GNU/ out of it.

    That said, Nice job! I love to see the capabilities of Linux expanded in new directions like this. Cool work. I wish I had time to work on cool projects like that.

    --
    set softtabstop=4 shiftwidth=4 expandtab nocp worlddomination
  3. I hope the meta-data performance improved... by Ayanami+Rei · · Score: 2, Interesting

    I found that gigabit NFS was usually much faster with files smaller than 1MB. I guess because either way, you still had to go through one server to set up each FS operation. NFS had been around longer; the Sun implementation was hard to beat.

    Has the meta-data server been speed up at all, or made distributed with some kind of coherency-syncro backend?

    --
    THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
    1. Re:I hope the meta-data performance improved... by alecthomas · · Score: 2, Informative
      Has the meta-data server been speed up at all, or made distributed with some kind of coherency-syncro backend?

      From the PVFS2 Guide:

      The new design has a number of important features, including:

      * modular networking and storage subsystems,
      * powerful request format for structured non-contiguous accesses,
      * flexible and extensible data distribution modules,
      * distributed metadata,
      * stateless servers and clients (no locking subsystem),
      * explicit concurrency support,
      * tunable semantics,
      * flexible mapping from file references to servers,
      * tight MPI-IO integration, and
      * support for data and metadata redundancy.
    2. Re:I hope the meta-data performance improved... by rizzy · · Score: 3, Informative

      > * flexible and extensible data distribution modules,
      > * distributed metadata,
      > * stateless servers and clients (no locking subsystem),

      Just to clarify... while we have distributed metadata, we don't have *replicated* metadata. At least, not yet.

      If you have multiple metadata servers they will do load balancing. If you are working with lots and lots of small files, having a couple metadata servers might alieviate a possible bottleneck.

  4. Re:I know this is for large clusters..... by brsmith4 · · Score: 3, Informative

    Simple answer: No. This is commonly used for allocated scratch space in cluster environments e.g. beowulf. We use it to reduce the reads and writes that usually bring an NFS system to its knees. It would not help Bittorrent.

  5. Re:Redundancy ? by REggert · · Score: 2, Interesting

    I use Andrew File System (specifically, http://www.openafs.org/) for my files, since I was used to using it at school, and I'm fond of its access control system. It allows you to designate redudant sites for your volumes for backup or load balancing purposes. However, its major downside is that it's optimized for reads but not for writes (PVFS would probably work better if you need optimal write performance), and it can be a real bitch to set up for the first time. I've also yet to figure out how to get it to work through my NAT, though it's supposed to be possible. It beats the hell out of NFS (v2, at least, I haven't really taken a look at NFS v3) in terms of reliability, security, and scalability, though.

    --

    cp /dev/zero ~/signature.txt