Ask Slashdot: User-Friendly, Version-Preserving File Sharing For Linux?

petherfile writes: I've been a professional with Microsoft stuff for more than 10 years and I'm a bit sick of it to be honest. The one that's got me stuck is really not where I expected it to be. You can use a combination of DFS and VSS to create a file share where users can put whatever files they are working on that is both redundant and has "previous versions" of files they can recover. That is, users have a highly available network location where they can "go back" to how their file was an hour ago. How do you do that with Linux?

This is a highly desirable situation for users. I know there are nice document management things out there that make sharepoint look silly, but I just want a simple file share, not a document management utility. I've found versioning file systems for Linux that do what Microsoft does with VSS so much better (for having previous version of files available.) I've found distributed file systems for Linux that make DFS look like a bad joke. Unfortunately, they seem to be mutually exclusive. Is there something simple I have missed?

Comments

FreeNAS

[Anonymous Coward]

FreeNAS will do all of that shiny stuff. And snapshots, too.

Re:FreeNAS

[Tailhook]

Yes, FreeNAS will get you there. Since versioning is a key requirement you will want to use ZFS. The thing you need for that is a plenty of RAM. It's not just a performance concern. ZFS can be unstable if not fed enough RAM.

So budget for something with a lot of installed RAM on day one, and some room to grow as you add more storage.

Yes, FreeNAS isn't Linux. The simple fact is that Linux has so far failed to achieve parity with other systems, both contemporary and historical, that provide advanced file system features. BTRFS might get there one day. ZFS is persona non grata. LVM can serve some of your expectations, but not all.

So look beyond Linux. In addition to FreeNAS there is proprietary stuff; they still make NetApps and they still work as good as ever. Dell has EqualLogic boxes that will snapshot volumes all day long. If you have the dosh there are all sort of solutions. If you're dosh-challenged then look to FreeNAS.

ZFS

[blowfly7012]

Look at ZFS. Supports snapshots, SMB and NFS. And you can show the snapshots as a read-only directory to users.

VMS

[FranTaylor]

use VMS, it's built in

TOPS-20 had it too

yearning already for the lost technology of the 1970s

OwnCloud

[jd142]

Take a look at the features list at https://owncloud.org/features/. It seems to have what you want. I played with it a couple of years ago and it was easy to set up then. Unfortunately I never had the opportunity to try it in production.

WebDAV

[Anonymous Coward]

Search Google for WebDAV auto-versioning.

I have set up (and used for many years) a WebDAV file share served by Apache (with an SVN backend). It can be used as an SVN repository (with checkin comments, etc.) or used as a simple remote file share that automatically creates revisions for the changes. I have used various WebDAV clients (built in to Linux, Mac, Windows) to access and modify the contents of the files.

Hope that gives you another area to explore.

Re:WebDAV

[flink]

Assuming you already have svn installed and copy of Apache httpd with mod_dav_svn and you are are running on some flavor of *NIX.

Create an svn repository somewhere, eg.

# svnadmin create /opt/svn-repo

Create a password for your repo, replace username and password as appropriate

# htpasswd /opt/svn-repo/conf/htpasswd USERNAME PASSWORD
# chmod 640 /opt/svn-repo/conf/htpasswd

Fix the permissions of the repo so that the user that httpd runs as can write to the repo database. Replace www with whatever the appropriate user is:

# cd /opt/svn-repo
# chgrp -R www .
# chmod -R g+r .
# chmod -R g+rwX db locks
# find db locks -type d -exec chmod g+s '{}' ';'

Open httpd.conf and add/uncomment the following lines in the LoadModule section:

LoadModule dav_module modules/mod_dav.so
LoadModule dav_svn_module modules/mod_dav_svn.so

At the very bottom of your httpd, add a location for your repository:

<Location /repo>
    DAV svn
    SVNPath /opt/svn-repo
    SVNAutoversioning on

    AuthType Basic
    AuthName "Subversion repository"
    AuthUserFile /opt/svn-repo/conf/htpasswd

    Require valid-user
</Location>

Restart apache and then test your config:

    # svn ls http://localhost/repo --username USERNAME --password PASSWORD --no-auth-cache
    #

No errors means everything is working.

See the manual for instructions on mounting the WebDAV share with various clients. Note that Windows is kind of problematic for this out of the box and you may need to use a third party file system driver such as NetDrive.

Simething simple you missed?

[Orgasmatron]

rsync with the --compare-dest option will give changed files, and --link-dest will give whole file trees at set times.

You can do it pretty simple, or quite complicated, depending on your needs and preferences.

rsync --link-dest makes a new tree with the current time, using only enough space for the directory tree and the changed files. On my box, I use it in a cron that runs every 5 minutes and cycles through my backup list. If any of them are older than the interval, it fires off the backup script specific to that type of connection (local LVM, nfs, CIFS/SMB, ssh, etc).

A second cron then prunes those directories so that I've got fewer copies as I go back in time. An example would be pulling a copy every 15 minutes and keep every copy for 2 weeks, keep one from each hour for a month, one per day for a year, one per month for 10 years, and one per year forever.

This can be easily adapted to other schemes. --compare-dest will make a tree with just the changed files, which you can then gather up and sort into the archival tree. Run a second (plain) rsync to sync up the comparison directory when done.

Re:Simething simple you missed?

[bartle]

I would also recommend looking at rsnapshot which is built on top of rsync.

I used to use a development system where the entire fire tree was mirrored at the top of every hour. Recovering old files was as simple as navigating to a different directory.

Personally, I like the rsync solution because it is filesystem agnostic. It also has been around for a long time; whatever you're trying to do, I can guarantee that someone was doing it with rsync 20 years ago.

There are a few options.

[mr_mischief]

Contrary to popular rumors, there are a number of ways to do what you want. I can't vouch for all of these combinations working and wouldn't be too optimistic about tackling some of them. The more advanced stuff can take quite a while to ramp up to speed.

If you don't mind FUSE as an intermediary, there's gitfs that uses git as a file system (which is kind of is anyway, beyond being just a VCS). It creates a new version on every file close. You can point it to a git remote on the same machine or across a network which lives on any filesystem.

You already found that there are some non-mainline kernel modules for filesystems like next3, ext3cow, or tux3 that do versioning on write. NILFS is actually in the kernel these days (since 2.6.something) . More information about NILFS2 shows that it's somewhat slow but that it is in fact a stable, dependable file system.

Subversion has a feature that you can put WebDAV in front of it, mount the WebDAV as a filesystem somewhere, and every write creates a new revision of the file in SVN. That gets you networked and versioned. This works similarly to gitfs but uses WebDAV. You could if you wanted use dav2fs in front of that to treat it like a normal file system again.

You can then share any of these over SMB with Samba. Or you can shared them via NFS.

If you need really high-end, fast, replicated network filesystems you can use any of the clustered storage systems that will use a storage node's underlying files with any of these below that, but that will put your revisions underneath everything else rather than on top. Then there's using something like gitfs with the remote on top of, for example, DRDB, XtreemFS, or Ceph (for example even across CephFS which presents Ceph as a normal POSIX filesystem). This latter option puts your revisions closer to the user and then each revision gets replicated.

I've personally never used some of the more exotic combinations listed here. You could in theory put NILFS2 on LVM with DRBD as the physical layer (since DRBD supports that) and then serve that file system via Samba (CIFS) or NFS which I would expect to work well enough if slowly.

Re:sharepoint...

[thegarbz]

You must not be using it right. Sharepoint can do EVERYTHING. It is a one stop shop for all your worlds problems. It has a feature list that make most OSes look lame in comparison. Just think of all the use cases:

- You want document management with integration into popular MS products which breaks horribly whenever you use something non-standard? SHAREPOINT!
- You want a metadata search system which fails to properly search metadata fields? SHAREPOINT!
- You want to run a website with dynamic content that is so difficult to edit you'll yearn for using HTML and Notepad? SHAREPOINT!
- Are your users being too productive? Want to slow people down a notch with slow load times so they don't embarrass you? SHAREPOINT!
- How about an integrated user management system which allows you to search users by hitting a single key and waiting half an hour! SHAREPOINT!

And on the days when you think you have hit rock bottom, your corporation is slowly falling apart after a SAP implementation has screwed the accounts system so badly that few invoices can be found, we have 3 words for you that you can integrate and ensure that the invoice you do find become unusable: SHARE FUCKING POINT!

Sharepoint is a great OS, all it needs is a decent text editor. Oh wait, MS Word. ... SHAREPOINT!