Slashdot Mirror

Contrary to popular rumors, there are a number of ways to do what you want. I can't vouch for all of these combinations working and wouldn't be too optimistic about tackling some of them. The more advanced stuff can take quite a while to ramp up to speed.

If you don't mind FUSE as an intermediary, there's gitfs that uses git as a file system (which is kind of is anyway, beyond being just a VCS). It creates a new version on every file close. You can point it to a git remote on the same machine or across a network which lives on any filesystem.

You already found that there are some non-mainline kernel modules for filesystems like next3, ext3cow, or tux3 that do versioning on write. NILFS is actually in the kernel these days (since 2.6.something) . More information about NILFS2 shows that it's somewhat slow but that it is in fact a stable, dependable file system.

Subversion has a feature that you can put WebDAV in front of it, mount the WebDAV as a filesystem somewhere, and every write creates a new revision of the file in SVN. That gets you networked and versioned. This works similarly to gitfs but uses WebDAV. You could if you wanted use dav2fs in front of that to treat it like a normal file system again.

You can then share any of these over SMB with Samba. Or you can shared them via NFS.

If you need really high-end, fast, replicated network filesystems you can use any of the clustered storage systems that will use a storage node's underlying files with any of these below that, but that will put your revisions underneath everything else rather than on top. Then there's using something like gitfs with the remote on top of, for example, DRDB, XtreemFS, or Ceph (for example even across CephFS which presents Ceph as a normal POSIX filesystem). This latter option puts your revisions closer to the user and then each revision gets replicated.

I've personally never used some of the more exotic combinations listed here. You could in theory put NILFS2 on LVM with DRBD as the physical layer (since DRBD supports that) and then serve that file system via Samba (CIFS) or NFS which I would expect to work well enough if slowly.

File revisions
Many early operating systems could keep several versions of a file. This was in UNIVAC EXEC-8 (now OS-2200 and still in use) in 1967. Creating a new empty file and then writing it did not make the file visible to other processes until the file was closed and committed. The new file then became the latest version, the old file became the previous version, and if a retention limit was specified and had been reached, the oldest version was deleted. UNIX/Linux/DOS/Windows pathname-based systems don't do that, and so atomic file replacement tends to be difficult, non-portable, and often not done.

Have you tried modern versioning filesystems as NILFS?

nilfs2 on linux is an filesystem designed just for flash memory. It was introduced mainline as part of 2.6.30. It'll be a while before it becomes usable though.

http://nilfs.org/

Try out different filesystems, NILFS seems to be optimized for FLASH usage.

Brtfs could also be worth a try.

use the "noop" IO/Scheduler with nilfs:

echo noop > /sys/block/sdX/queue/scheduler

Postmark benchs on an usb-stick (shameless copied from here:

ext3 (mount -o noatime,noadirtime, normale Partition, scheduler cfq): 49 Transactions/s

nilfs2 (Partition aligned 128k, scheduler noop, protection_period 10s): 588 Transactions/s

if those test results are consistent, ditto that. but without strong linux distro support (i.e. ubuntu) one would have to be an experienced poweruser (esp. with driver/fs configuration, scripts, file permissions, etc.) to implement it (either than or know one such poweruser. Although I haven't used linux for too long (only ~3 years) I think it's safe to assume that configuration of that level would require more than experience in following how-to's, wikis, man pages, and other documentation-- it would require some serious experience, at least to do properly.

Try out different filesystems, NILFS seems to be optimized for FLASH usage.

Brtfs could also be worth a try.

use the "noop" IO/Scheduler with nilfs:
echo noop > /sys/block/sdX/queue/scheduler

Postmark benchs on an usb-stick (shameless copied from here:
ext3 (mount -o noatime,noadirtime, normale Partition, scheduler cfq): 49 Transactions/s
nilfs2 (Partition aligned 128k, scheduler noop, protection_period 10s): 588 Transactions/s

NILFS is a log-structured file system developed for the Linux, and it is downloadable on this site as open-source software.

Because old stuff can be overwritten when you fill the whole disk. As I mentioned in other posting, data writes are Real Fast in log filesystems, but data reads are Real Slow.

The biggest problem of this filesystem (link is missing from the original posting) is that it's Not Really Ready (among other important stuff, mmap() is not implemented yet).

http://www.nilfs.org/

The system might hang under heavy load.

The system hangs on a disk full condition.
Aren't those kind of important to saying that something is stable?