Slashdot Mirror

half as many servers to hold the disks with half of the redundancy.

Now this is cool still need mon servers

http://ceph.com/community/500-...

Well, if you want frequent filesystem or directory snapshots rather than per-file per-change versioning then your options are much broader.

LVM can do snapshots besides doing disk pooling, and can do a sort of clustering. Btrfs pools disks and does snapshots without the clustering. XFS has indirect support for snapshots -- it allows one to freeze the filesystem and snapshot using the volume manager. Ceph is a highly available clustered storage system with a POSIX FS face and does snapshots. Lustre allows taking LVM snapshots of its MDT and OST filesystems, although doing so often could be a bottleneck. Gluster (since 3.7 or so) actually wraps around LVM's snapshot tools so long as each "brick" is on a thin LVM that contains no data besides the brick.

ZFS does pooling and snapshots. It's supported-ish on Linux both as a kernel module and as a FUSE filesystem. It has much better support on the BSDs and of course on OpenSolaris/Solaris.

It's notable that the Ceph folks recommend Btrfs underneath Ceph as the local filesystem for development and testing but XFS under it (or optionally ext4) for production at the moment.

Contrary to popular rumors, there are a number of ways to do what you want. I can't vouch for all of these combinations working and wouldn't be too optimistic about tackling some of them. The more advanced stuff can take quite a while to ramp up to speed.

If you don't mind FUSE as an intermediary, there's gitfs that uses git as a file system (which is kind of is anyway, beyond being just a VCS). It creates a new version on every file close. You can point it to a git remote on the same machine or across a network which lives on any filesystem.

You already found that there are some non-mainline kernel modules for filesystems like next3, ext3cow, or tux3 that do versioning on write. NILFS is actually in the kernel these days (since 2.6.something) . More information about NILFS2 shows that it's somewhat slow but that it is in fact a stable, dependable file system.

Subversion has a feature that you can put WebDAV in front of it, mount the WebDAV as a filesystem somewhere, and every write creates a new revision of the file in SVN. That gets you networked and versioned. This works similarly to gitfs but uses WebDAV. You could if you wanted use dav2fs in front of that to treat it like a normal file system again.

You can then share any of these over SMB with Samba. Or you can shared them via NFS.

If you need really high-end, fast, replicated network filesystems you can use any of the clustered storage systems that will use a storage node's underlying files with any of these below that, but that will put your revisions underneath everything else rather than on top. Then there's using something like gitfs with the remote on top of, for example, DRDB, XtreemFS, or Ceph (for example even across CephFS which presents Ceph as a normal POSIX filesystem). This latter option puts your revisions closer to the user and then each revision gets replicated.

I've personally never used some of the more exotic combinations listed here. You could in theory put NILFS2 on LVM with DRBD as the physical layer (since DRBD supports that) and then serve that file system via Samba (CIFS) or NFS which I would expect to work well enough if slowly.

See this thread:

http://lists.ceph.com/pipermai...

It has the ability to do RBD Snapshots, it's not a perfect solution but it does work. We're actually in the process of building a Ceph system for testing of climate modeling data. https://ceph.com/community/blo...

Dear Shuttleworth,

Don't think I don't know what you are doing. It was clever of you to have invested $1M in Inktank to support Ceph. That got you a lot of hits on ceph.com. It may take me all year, but through the power of science I will eventually beat your record of who can drive more traffic to our website. Like a master ninja I will blind you with my amazing insights. Just look at my analysis of Ceph's write performance on different disk controllers. Yeah it didn't get as many hits as your little investment announcement, but this is just the beginning. So my questions is, do you want to just give up now?

Mark

(Nelson, not Shuttleworth!)

• * Native API
• * Via a RESTful interface that can handle native Amazon S3 and Swift API calls
• * As a thinly provisioned block device
• * Mount it as a POSIX-compliant file system via CephFS (although this is a bit rough for production environments just yet)

Josh Durgin has actually done some really interesting work in using the block device (RBD) to back Cinder which you can read a bit about here.

The cool part about Ceph is it was designed to be massively scalable (petabytes and beyond) and extremely fault tolerant / HA / etc. DreamHost actually just built out a huge production deployment of Ceph and OpenStack for their new DreamCompute / DreamObjects offering. If you have questions feel free to hit up the #Ceph irc channel at irc.oftc.net or poke me via email (my UN at inktank.com) and I'll see if I can't find the right person to help.

OpenStack really has some awesome potential, and we're excited about poking at it more with our semi-sharp Ceph-stick. Good luck.

• * Native API
• * Via a RESTful interface that can handle native Amazon S3 and Swift API calls
• * As a thinly provisioned block device
• * Mount it as a POSIX-compliant file system via CephFS (although this is a bit rough for production environments just yet)

Josh Durgin has actually done some really interesting work in using the block device (RBD) to back Cinder which you can read a bit about here.

The cool part about Ceph is it was designed to be massively scalable (petabytes and beyond) and extremely fault tolerant / HA / etc. DreamHost actually just built out a huge production deployment of Ceph and OpenStack for their new DreamCompute / DreamObjects offering. If you have questions feel free to hit up the #Ceph irc channel at irc.oftc.net or poke me via email (my UN at inktank.com) and I'll see if I can't find the right person to help.

OpenStack really has some awesome potential, and we're excited about poking at it more with our semi-sharp Ceph-stick. Good luck.

Now that Ceph is gathering momentum since having been included in the mainline kernel, what other storage (or low level) advancements do you see on the horizon?

(full disclosure: I work for Inktank now, the consulting/services company that employs most of the core Ceph engineers)

Now that Ceph is gathering momentum since having been included in the mainline kernel, what other storage (or low level) advancements do you see on the horizon?

(full disclosure: I work for Inktank now, the consulting/services company that employs most of the core Ceph engineers)

Have you considered using a distributed file system such as ceph?

You will need more drives as the data takes twice the space, but on the other hand you won't need to worry about boot times or scalability anymore.