Database File System

ozy writes "With all the fuss about searching and Spotlight and WinFS, check out the Database File System a completely different interface for your files, implemented in KDE. There is actually a request for developers to join a project to implement this under GNOME and leave how we use the desktop today behind."

Comments

Backups, and being organized in a general way?

[manavendra]

...I can kind of see this would make it easy to search and locate documents. What about backups though? How would a user be able to group (manually) related files together, so that the whole bunch can be backed up later, without having to search for all seeminly related (or unrelated) keywords to trace all hitherto-unrelated documents?

Secondly, with this mass of files being spread over several disks, surely, this is in a way forcing the user to "search" for everything. Or isnt it? Will the underlying FS layer still be accessible in the general way that it is?

Version control would be nice as well

[zero-one]

I have always thought that version control (file histories, branching and atomic changes) would be nice to have at the file system level. Instead of storing myessay-firstdraft.doc, myessay-seconddraft.doc, myessay-final.doc, the file system should do the work. Then if I want to make a bunch of changes (perhaps I want to try a new page layout), I should be able to commit them as one atomic change (or throw them all away if I change my mind). Then, when I want to make a set of documents with US spelling, I should be able to branch the whole lot (using no disk space) and make the small changes from UK spelling while still being able to integrate other changes I have made.

Re:Version control would be nice as well

[ultrabot]

I have always thought that version control (file histories, branching and atomic changes) would be nice to have at the file system level.

Sounds like a job for an SVN plugin for Reiser4 file system. Anyone doing one already?

Reiserfs, storage and why do you want this?

[ACK!!]

Perhaps I am more organized than most but I already categorize my files and such in the hierachal file structure.

Isn't Rieserfs planning to do this on the kernel level?

Where does that leave other fs choices and storage and other idea dbfs?

I see more and more people saying look what neat things you can do with these tools.

But really why do you personally want something like this?

Curious to see the response is all.

Re:Reiserfs, storage and why do you want this?

[BenjyD]

The problems with the hierachical system are:

- maintenance overhead on part of the user to create hierachy and maintain it. Every time you save a file you have to think "where do I put this?"
- Finding files can be hard. Is that letter about the planning application in Documents/Letters or Documents/Planning App?
- keeping files in two or more places at once is hard (as in the previous example). You can use softlinks, but that's hardly ideal and doesn't survive moving things around.

Basically, the current file system imposes a significant overhead. Most power users have restructured the way they work and use a computer in order to minimise that overhead without really noticing. It's just become one of those things you have to do, like remembering to save documents.

Why not shift the burden of organising the files onto the enormously powerful computer, rather than take up valuable human mental resources.

Re:Reiserfs, storage and why do you want this?

[ctr2sprt]

OK, let's consider an example other than documents. Joe is a big music fan with a couple hundred CDs. He likes having instant access to any one song when he wants it and has a lot of hard drive space, so he's MP3-ized (or ogg-ized or flac-ized or whatever) his entire collection, plus all the MP3s he's acquired via other means.

Joe is pretty good about organization, so he names every MP3 properly with the group, album, and track names, plus the track number. (Something like The Beatles/White Album/01-Back in the USSR.mp3.) This way if he knows, for example, the track name but not what album it's on, he can find it pretty quickly using a method like yours.

The problem is if he wants to do something like put all the country music he has, for example, in a playlist. How does he do that? It can be done, certainly, but if he has a collection with several thousand MP3s it's so tedious and difficult as to be effectively impossible. What if he wants to listen to 60s rock? What if he wants to find a particular song he has, but all he knows is it's between 3 and 5 minutes long, came out between 2002 and 2003, and is probably categorized as either "pop" or "alternative?" What if he just wants a list of all the songs he never listens to because he's sick of what he's been playing lately? Or maybe he needs to free up disk space and wants to find out what he'll miss the least.

All these things are impossible to do in an efficient and timely manner using our current system. He can certainly use a command-line ID3 tagger to strip out the things he cares about, something like

find mp3 -type f -print0 | xargs -0 id3tag -l | grep 'Genre: Country'

but that's painfully slow: a second or two per file means a big connection will take 15 minutes or longer to scan, and if you typed "Gerne" by mistake you have to do it all over.

Now if you had a filesystem-like object which could be smart and store ID3 metadata in the filesystem, then it would be much faster: the main overhead to using the find/xargs/id3tag/grep approach described above comes from having to seek through the MP3 file to get at the metadata. The reason this needs to be a "core OS component," perhaps even part of the kernel, is that MP3 tags can change at whim and the filesystem needs to know about it or its metadata can get out of sync. It's possible, but impractical, to update this on a periodic basis, like the locatedb; it makes much more sense to have the kernel inform some plugin "Hey, this file just changed, do you care?" And if the plugin does care, it can look at the changes, see if it's affected, and possibly update the metadata to match. It could also go the other way, where Joe updates the filesystem metadata and the OS knows to update the MP3's ID3 tags too.

Comments by the Author

[data1]

The author is asking for help to move the project to Gnome.

Quote:There is of-course the hard choice of platform. I choose KDE? because I am familiar with QT a bit, and because it is inherently object-oriented, being C++ and all. But in my mind GNOME? is much closer to how I would like a desktop system to function. So I would like to go for the GNOME option. I leave KDE developers to do what they want with this, and I am offering them support. But I would like to focus my efforts on GNOME and implementing the above in GNOME.

Any volunteers?
In the first place we will need developers. Would you like to join, send me an email (o.gorter@student.utwente.nl) with DBFS and JOIN somewhere in the subject. If you are not a developer, but still would like to help, please revisit this page in a few weeks. There will probably be a community website by then somewhere.

Sigh, Andrew Morton seemed to be right...

[greppling]

...when he said on LKML, slightly paraphrased: "The only reason I see to put filesystem semantic enhancements into the kernel, is that it would be socially hard to get people to agree an a single userspace library."

(In the course of the heated discussion about Reiser4.)

This looks like BLINKX plus more

[lcsjk]

I already use blinkx, beta, from http://www.blinkx.com/, to automatically search my files along with internet keywords. It doesn't have the search by date or extension and is not configurable to my liking, but it seems to do a good job of finding things I have misplaced. Integrated with the author's system, this could make a great search system.

Normally I file things in a hierarchial method by year and month and by project name (2004file/9sep/) or (2004file/workfile/projectname), but still I lose track now and then and need keywords. Change the "slash" slant to fit your OS.

What About Code Bloat?

[reporter]

These days, operating systems like both Linux and Windows XP have too many bells and whistles that I simply do not need. Unfortunately, these bells and whistles drastically increase the amount of space that I need on my hard drive.

Adding a database layer to Gnome sounds like using another 300 megabytes of storage on my hard drive. I simply do not need the database.

If the FSF/GNU folks really want to do something revolutionary, they should fork Linux+Gnome into 2 distinct paths: minimalist and maximalist. The maximalist is what we have now. The minimalist is a minimally featured Linus+Gnome distribution. It is the bare minimum in functionality that we need to have a decent operating system and desktop.

Into this minimalist installation, I will then add the applications (e.g. MatLab) that I use daily.

standard filesystems are NOT databases

[lkcl]

standard filesystems only have ONE index - a hierarchical one that contains a certain amount of real-time-updated indexing (such as the timestamps on a directory)

but it is NOT a relational database: you CANNOT easily create or use an alternative index to your files.

that's what all the fuss is about.

some people mentioned here that they already organise their files. great. fantastic.

HOW LONG DID IT TAKE YOU?

and how long would it take to reorganise?

with a relational database, all your indexes are updated AUTOMATICALLY.

therefore, doing searches on a relational database filesystem (find me all music files with dates between last week and last month: SELECT * from files WHERE files.type = "music" and files.date NOW() - 7days

you _can't_ do that sort of thing on a traditional filesystem.

sure, you can emulate it by creating symbolic links all over the place, but what happens when a file is deleted or moved? you need to manage / relocate the symbolic links...

nah.

databases.

fantastic idea.

now can we have them as a kernel module, pleeease?

Read the article, read some history

[eschasi]

To quote from the article (which most folks have not read, as usual):

The DBFS does not actually store files, it holds references to files on the underlying hierarchy based file system.

That line alone should answer many of the questions re backup, speed of FS performance, etc.

At a deeper technical level, nany of the questions asked here have historical answers or clues in The Design and Implementation of the Inversions File System. The abstract reads:

This paper describes the design, implementation, and performance of the Inversion file system. Inversion provides a rich set of services to file system users, and manages a large tertiary data store. Inversion is built on top of the POSTGRES database system, and takes advantage of low-level DBMS services to provide transaction protection, fine-grained time travel, and fast crash recovery for user files and file system metadata. Inversion gets between 30% and 80% of the throughput of ULTRIX NFS backed by a non-volatile RAM cache. In addition, Inversion allows users to provide code for execution directly in the file system manager, yielding performance as much as seven times better than that of ULTRIX NFS.

Note that this paper was published in early 1993. Many of the issues it addresses are relevant to DBFS, and many of DBFS's advantages are foreseen by that paper. IMHO DBFS has chosen a direction that should have better performance than inversion, not to mention lower risk and easier failure recovery.

Inversion was built on POSTGRES, which makes one wonder what happened to the source.

Not Unix

[flossie]

the DBFS does not store system files: No shared libraries, no font files or others like that. These are not documents, not files you look up at a day to day basis, and have no place in a file system.

Whether or not you look at system files every day probably depends on what you are doing with your machine and what you consider "system files" to be. Moreover, this idea would seem to go entirely against the whole UNIX "everything is a file" philosophy which is supposed to be one of the great strengths of UNIX.

SVN + DBFS + 10TB Nano Hoozie

[KrackHouse]

I'm using Subversion for a project and the idea of Atomic Commits seems like an obvious direction for file systems. If that other slashdot story is correct, storage becomes less of an issue and it would be possible to roll back the system to any point in time or to only roll back one file if need be. Now throw an intuitive way to navigate files on top of that and you've got a sure winner.

In the grand scheme of things, only a very small handful of us on earth are aware of Linux or even know what an Operating System is for that matter. File systems seem to be the big stumbling block for new users. Anything that can make computers and therefore access to information easier for the coming waves of new computer users (maybe billions of people?) will be a good thing. Even if the "bloat" slows down the system by 10%.

I hate to preach but that old quote comes to mind "With great power comes great responsibility". I don't think most of the people working on the OS that will soon dominate in developing nations (that's Linux) are aware of the harm they can do by slowing down Linux development with petty personal disputes. Like it or not, Linux is no longer just an edgy hacker tool. It has the potential to change the lives of Billions of people.

Re:gnome people...

[Anonymous Coward]

Here is my opposite hierarchical "document oriented" desktop: it does not have to be based on database file system, but intelligent caching and indexing

1) No "Start" application menu. You never "start" an application. You always open (read:look at) a document by clicking on it. You want to write a new one? you click on "template" document, it immediately asks you for new name (there never exists unnamed one, even on memory - because there is no difference between memory and disk). Do you want to read your email? you look inside mail/inbox dirextory. Everything has a directory/file structure! The same is with web - although you can see (list) only "directories" (sites) where you have been. Otherwise you have to either type URL location dialog or use bookmark. Bookmarks kook just like links in ~/bookmarks/whatever_link_page. The same page you can list in virtual http://whatever.com/ directory (of course only if if you allready bookmarked or visited it)

2) No open/save dialog. "invisible" application flushes updates to the ducument frequently, it appears that you directly instantly to the file. On the other hand you should be able to see a "historic" (you can say undoed) under some history.

3) No application "Close" (x, exit). You just remove (minimise) the document from your view. If you do not look at the document (work with, view url, ...) for some time, intelligent caching system invisibly closes application (of course the document is flushed, probably quite some time ago) and frees resource to the system. There is no difference between memory and disc allocated documents from user point of view.

4) Taskbar is not a list of running applications but a "History" list of last opened documents (URLs, ...) with more-less constant number if items

5) Window name functions also as "location" (URL) input, saves space

6) there is no "all time" visible menu, toolbar, location bar (see (5)) inside the window - those only take space. Right mouse button opens context menu, together with Alt it opens "document" menu (full-blown what is more-less now visible on the top of windows), with Ctrl it opens "shortcut" menu (ala toolbar) under cursor (so you do not need to move too far.

Oh, and did I say no start/close application?

Roman Kantor

This is not a new idea.....

[Cnik70]

Take a look at an AS/400 (iSeries). They've been around for more than 10 years. And before that you had the System 38 and System 36... and so on..... Why is this some big revelation now?