Database File System

ozy writes "With all the fuss about searching and Spotlight and WinFS, check out the Database File System a completely different interface for your files, implemented in KDE. There is actually a request for developers to join a project to implement this under GNOME and leave how we use the desktop today behind."

Comments

Backups, and being organized in a general way?

[manavendra]

...I can kind of see this would make it easy to search and locate documents. What about backups though? How would a user be able to group (manually) related files together, so that the whole bunch can be backed up later, without having to search for all seeminly related (or unrelated) keywords to trace all hitherto-unrelated documents?

Secondly, with this mass of files being spread over several disks, surely, this is in a way forcing the user to "search" for everything. Or isnt it? Will the underlying FS layer still be accessible in the general way that it is?

Version control would be nice as well

[zero-one]

I have always thought that version control (file histories, branching and atomic changes) would be nice to have at the file system level. Instead of storing myessay-firstdraft.doc, myessay-seconddraft.doc, myessay-final.doc, the file system should do the work. Then if I want to make a bunch of changes (perhaps I want to try a new page layout), I should be able to commit them as one atomic change (or throw them all away if I change my mind). Then, when I want to make a set of documents with US spelling, I should be able to branch the whole lot (using no disk space) and make the small changes from UK spelling while still being able to integrate other changes I have made.

Re:Version control would be nice as well

[ultrabot]

I have always thought that version control (file histories, branching and atomic changes) would be nice to have at the file system level.

Sounds like a job for an SVN plugin for Reiser4 file system. Anyone doing one already?

Reiserfs, storage and why do you want this?

[ACK!!]

Perhaps I am more organized than most but I already categorize my files and such in the hierachal file structure.

Isn't Rieserfs planning to do this on the kernel level?

Where does that leave other fs choices and storage and other idea dbfs?

I see more and more people saying look what neat things you can do with these tools.

But really why do you personally want something like this?

Curious to see the response is all.

Re:Reiserfs, storage and why do you want this?

[ctr2sprt]

OK, let's consider an example other than documents. Joe is a big music fan with a couple hundred CDs. He likes having instant access to any one song when he wants it and has a lot of hard drive space, so he's MP3-ized (or ogg-ized or flac-ized or whatever) his entire collection, plus all the MP3s he's acquired via other means.

Joe is pretty good about organization, so he names every MP3 properly with the group, album, and track names, plus the track number. (Something like The Beatles/White Album/01-Back in the USSR.mp3.) This way if he knows, for example, the track name but not what album it's on, he can find it pretty quickly using a method like yours.

The problem is if he wants to do something like put all the country music he has, for example, in a playlist. How does he do that? It can be done, certainly, but if he has a collection with several thousand MP3s it's so tedious and difficult as to be effectively impossible. What if he wants to listen to 60s rock? What if he wants to find a particular song he has, but all he knows is it's between 3 and 5 minutes long, came out between 2002 and 2003, and is probably categorized as either "pop" or "alternative?" What if he just wants a list of all the songs he never listens to because he's sick of what he's been playing lately? Or maybe he needs to free up disk space and wants to find out what he'll miss the least.

All these things are impossible to do in an efficient and timely manner using our current system. He can certainly use a command-line ID3 tagger to strip out the things he cares about, something like

find mp3 -type f -print0 | xargs -0 id3tag -l | grep 'Genre: Country'

but that's painfully slow: a second or two per file means a big connection will take 15 minutes or longer to scan, and if you typed "Gerne" by mistake you have to do it all over.

Now if you had a filesystem-like object which could be smart and store ID3 metadata in the filesystem, then it would be much faster: the main overhead to using the find/xargs/id3tag/grep approach described above comes from having to seek through the MP3 file to get at the metadata. The reason this needs to be a "core OS component," perhaps even part of the kernel, is that MP3 tags can change at whim and the filesystem needs to know about it or its metadata can get out of sync. It's possible, but impractical, to update this on a periodic basis, like the locatedb; it makes much more sense to have the kernel inform some plugin "Hey, this file just changed, do you care?" And if the plugin does care, it can look at the changes, see if it's affected, and possibly update the metadata to match. It could also go the other way, where Joe updates the filesystem metadata and the OS knows to update the MP3's ID3 tags too.

Comments by the Author

[data1]

The author is asking for help to move the project to Gnome.

Quote:There is of-course the hard choice of platform. I choose KDE? because I am familiar with QT a bit, and because it is inherently object-oriented, being C++ and all. But in my mind GNOME? is much closer to how I would like a desktop system to function. So I would like to go for the GNOME option. I leave KDE developers to do what they want with this, and I am offering them support. But I would like to focus my efforts on GNOME and implementing the above in GNOME.

Any volunteers?
In the first place we will need developers. Would you like to join, send me an email (o.gorter@student.utwente.nl) with DBFS and JOIN somewhere in the subject. If you are not a developer, but still would like to help, please revisit this page in a few weeks. There will probably be a community website by then somewhere.

Sigh, Andrew Morton seemed to be right...

[greppling]

...when he said on LKML, slightly paraphrased: "The only reason I see to put filesystem semantic enhancements into the kernel, is that it would be socially hard to get people to agree an a single userspace library."

(In the course of the heated discussion about Reiser4.)

This looks like BLINKX plus more

[lcsjk]

I already use blinkx, beta, from http://www.blinkx.com/, to automatically search my files along with internet keywords. It doesn't have the search by date or extension and is not configurable to my liking, but it seems to do a good job of finding things I have misplaced. Integrated with the author's system, this could make a great search system.

Normally I file things in a hierarchial method by year and month and by project name (2004file/9sep/) or (2004file/workfile/projectname), but still I lose track now and then and need keywords. Change the "slash" slant to fit your OS.

Read the article, read some history

[eschasi]

To quote from the article (which most folks have not read, as usual):

The DBFS does not actually store files, it holds references to files on the underlying hierarchy based file system.

That line alone should answer many of the questions re backup, speed of FS performance, etc.

At a deeper technical level, nany of the questions asked here have historical answers or clues in The Design and Implementation of the Inversions File System. The abstract reads:

This paper describes the design, implementation, and performance of the Inversion file system. Inversion provides a rich set of services to file system users, and manages a large tertiary data store. Inversion is built on top of the POSTGRES database system, and takes advantage of low-level DBMS services to provide transaction protection, fine-grained time travel, and fast crash recovery for user files and file system metadata. Inversion gets between 30% and 80% of the throughput of ULTRIX NFS backed by a non-volatile RAM cache. In addition, Inversion allows users to provide code for execution directly in the file system manager, yielding performance as much as seven times better than that of ULTRIX NFS.

Note that this paper was published in early 1993. Many of the issues it addresses are relevant to DBFS, and many of DBFS's advantages are foreseen by that paper. IMHO DBFS has chosen a direction that should have better performance than inversion, not to mention lower risk and easier failure recovery.

Inversion was built on POSTGRES, which makes one wonder what happened to the source.

Not Unix

[flossie]

the DBFS does not store system files: No shared libraries, no font files or others like that. These are not documents, not files you look up at a day to day basis, and have no place in a file system.

Whether or not you look at system files every day probably depends on what you are doing with your machine and what you consider "system files" to be. Moreover, this idea would seem to go entirely against the whole UNIX "everything is a file" philosophy which is supposed to be one of the great strengths of UNIX.

SVN + DBFS + 10TB Nano Hoozie

[KrackHouse]

I'm using Subversion for a project and the idea of Atomic Commits seems like an obvious direction for file systems. If that other slashdot story is correct, storage becomes less of an issue and it would be possible to roll back the system to any point in time or to only roll back one file if need be. Now throw an intuitive way to navigate files on top of that and you've got a sure winner.

In the grand scheme of things, only a very small handful of us on earth are aware of Linux or even know what an Operating System is for that matter. File systems seem to be the big stumbling block for new users. Anything that can make computers and therefore access to information easier for the coming waves of new computer users (maybe billions of people?) will be a good thing. Even if the "bloat" slows down the system by 10%.

I hate to preach but that old quote comes to mind "With great power comes great responsibility". I don't think most of the people working on the OS that will soon dominate in developing nations (that's Linux) are aware of the harm they can do by slowing down Linux development with petty personal disputes. Like it or not, Linux is no longer just an edgy hacker tool. It has the potential to change the lives of Billions of people.