Database File System

ozy writes "With all the fuss about searching and Spotlight and WinFS, check out the Database File System a completely different interface for your files, implemented in KDE. There is actually a request for developers to join a project to implement this under GNOME and leave how we use the desktop today behind."

Comments

gnome people...

[alphan]

...seems to have something more interesting: storage

Re:gnome people...

[Jellybob]

Storage has been more or less dead as far as I can see for a while now, however Beagle is showing good progress on the same front, having been demoed at conferences recently.

Re:gnome people...

[noselasd]

Not really still activities in the CVS
The current release (0.2) is just some proof of concept, devels
are working on a nice'n'real solution.

Re:gnome people...

[kwoff]

Beagle is written in C#? /me clicks the Back button

Re:gnome people...

[Anonymous Coward]

Here is my opposite hierarchical "document oriented" desktop: it does not have to be based on database file system, but intelligent caching and indexing

1) No "Start" application menu. You never "start" an application. You always open (read:look at) a document by clicking on it. You want to write a new one? you click on "template" document, it immediately asks you for new name (there never exists unnamed one, even on memory - because there is no difference between memory and disk). Do you want to read your email? you look inside mail/inbox dirextory. Everything has a directory/file structure! The same is with web - although you can see (list) only "directories" (sites) where you have been. Otherwise you have to either type URL location dialog or use bookmark. Bookmarks kook just like links in ~/bookmarks/whatever_link_page. The same page you can list in virtual http://whatever.com/ directory (of course only if if you allready bookmarked or visited it)

2) No open/save dialog. "invisible" application flushes updates to the ducument frequently, it appears that you directly instantly to the file. On the other hand you should be able to see a "historic" (you can say undoed) under some history.

3) No application "Close" (x, exit). You just remove (minimise) the document from your view. If you do not look at the document (work with, view url, ...) for some time, intelligent caching system invisibly closes application (of course the document is flushed, probably quite some time ago) and frees resource to the system. There is no difference between memory and disc allocated documents from user point of view.

4) Taskbar is not a list of running applications but a "History" list of last opened documents (URLs, ...) with more-less constant number if items

5) Window name functions also as "location" (URL) input, saves space

6) there is no "all time" visible menu, toolbar, location bar (see (5)) inside the window - those only take space. Right mouse button opens context menu, together with Alt it opens "document" menu (full-blown what is more-less now visible on the top of windows), with Ctrl it opens "shortcut" menu (ala toolbar) under cursor (so you do not need to move too far.

Oh, and did I say no start/close application?

Roman Kantor

Backups, and being organized in a general way?

[manavendra]

...I can kind of see this would make it easy to search and locate documents. What about backups though? How would a user be able to group (manually) related files together, so that the whole bunch can be backed up later, without having to search for all seeminly related (or unrelated) keywords to trace all hitherto-unrelated documents?

Secondly, with this mass of files being spread over several disks, surely, this is in a way forcing the user to "search" for everything. Or isnt it? Will the underlying FS layer still be accessible in the general way that it is?

Re:Backups, and being organized in a general way?

[manavendra]

Oh, and forgot to add that I wouldn't be happy if my disk begins to churn away as the "Crawler" updates its indexes, say, while I'm playing doom! Can't afford to have any other spooky, disk grinding noises, while squinting my way through those dark corridors..

Re:Backups, and being organized in a general way?

[carnivore302]

Will the underlying FS layer still be accessible in the general way that it is?
Sure, because the dbfs is implemented in userland. All applications will work as before, including ls.

I wonder if this could be made into a plugin for reiserfs 4.

Mark.

Re:Backups, and being organized in a general way?

[manavendra]

Well, you may find it funny, I've already messed up two pair of trousers - once when my phone rang and second when my friend put his hand on my shoulder to get my attention...

Re:Backups, and being organized in a general way?

[Aeiri]

You should have posted that as Anonymous Coward...

Re:Backups, and being organized in a general way?

[Ignominious Cow Herd]

Think of a directory structure as just one instance of a relational structure - a heirarchical or location-based one. Directories (or locations, or relationship to other files), even nested ones, are just another type of metadata. Once you have that concept in mind other things become obvious. For example you may group files by type (Image, Text) or by project, or by author. With a directory structure you either have to make multiple directories and symlink everything, or you're stuck with one view of your files.

In short it gives you multiple, simultaneous groupings of your data.

Re:Backups, and being organized in a general way?

[Pharmboy]

No, I think its funnier that he didn't. My guess is after he hit "submit" he went, "oh crap, now everyone knows I wear Depends when gaming".

Performance?

[saden1]

Isn't this thing with DB's getting a little excessive? You're adding another layer and step to storing data which will in all likely hinder performance. I'm not sure the benefit out weight the cost.

Re:Performance?

[Anonymous Coward]

Depends on what you are using your computer for of course.

You can say the same thing for a GUI, and its correct for certain applications of computers, but wrong in others.

Re:Performance?

[bogie]

Of course your right. But knowing that pretty smart people are working on this I don't think your going to see them go ahead with an implementation that's only half the speed of current linux file systems. I'm sure they'll only go ahead with this and integrate it into KDE when the performance is up to snuff. It's simply way too early to say that the benefits out weigh cost until the code is complete.

Re:Performance?

[psavo]

Isn't this thing with DB's getting a little excessive? You're adding another layer and step to storing data which will in all likely hinder performance. I'm not sure the benefit out weight the cost.

Well, if it's only a name-translation thingy, then it shouldn't affect performance of file reading (when operating on sufficiently big files), only file opening/stat:ing.

Re:Performance?

[kosmosik]

Nobody is sugesting to use such database FS for entire system. Only for specific data (f.e. user documents) - not entire system (binaries, libraries etc.) where such performance matters. Well in fact it will improve performance since right now applications that need such indexing (best examples are apps for organizing music (like iTunes) or digital pictures colections (like Adobe Photo Album or Google Picasa)) do it themselves which probably is not the fastest way and is not unified across the system. Now for *some* applications such view on files that lets you query for specific files/objects operating on query results rather as directories of files have much benefit. But it is only for organizing data, and in limited scope (as I've said - digital music, photography, probably some other fields). I don't really belive that this would seed up searching for office documents over LAN or smth. - when somebodys documents are in mess DB-FS won't change anything as the documents probably lack metadata, proper naming anyway.

Re:Performance?

[BenjyD]

Why not just run in console mode? All this GUI stuff is just getting in the way of absolute performance.

If it adds 0.5 seconds to every time you save a file, but saves you 20 seconds of filesystem navigating every time you open the file, that's a worthwhile tradeoff. Add to that the fact that copmuters don't get tired or bored, while humans do, and it makes even more sense to shift as much of the burden of working onto the computer as is practical.

Re:Performance?

[jgardn]

Not necessarily. Consider the performance of finding a document you wrote two years ago. How long does it take you to walk through the directory hierarchy browsing file names? How fast is the file search tool? Wouldn't it be faster if you could say "Show me the documents I wrote two years ago" and the refine the search or browse the results?

Storing data in a relational database is natural because it is more like the way we store data in our minds than the hierchical structures of traditional file systems.

Also, we allow a complete abstraction of the underlying database in relational systems. The database can store the data however it sees fit, and can arrange the data on disk without the users noticing a change.

I look forward to experimenting with a relational filesystem. I think it would be a wonderful thing to try out and see if it actually has the advantages I outlined above. I'd also like to see the actual disadvantages.

Re:Performance?

[kfg]

Why not just run in console mode? All this GUI stuff is just getting in the way of absolute performance.

Although this is a KDE related project the concept itself has nothing to do with whether you use a GUI or not and the performance hit comes at the level of the DB, not the GUI.

As for shifting the burden to the computer it doesn't really do much of that either as a human mind still has to formulate and input the query terms as well as judge the validity of the query result.

The DB as filesystem has a lot of merit, but really only in those situations where you have a massive number of files distributed across many systems. Take Google and the internet for example.

Now imagine having to google your local system to find every damned file.

Now, maybe I'm just different, but I've got 45,000 files spread across 1300 directories in my Home directory, and I can find any one of these by navigation in under your hypothetical 20 seconds saved, but then I'm the sort of person who sorts his laundry by placing it in seperate hampers in the first place instead of later spreading it across the basement floor and sorting it out. I know exactly where all my dirty whites are up front.

And the latter sort of person isn't likely to do a very good job of entering the metadata necessary to make a DB based filesystem work well anyway. The brain is a wonderful DB in and of itself and knows the "meaning" of things as well. I already know what 14t.jpg is all about. I'd have to tell my DB what it means.

Sure, let the computer do the work that it's better at, like recalculating spreadsheets or finding redheads with big ones on the web, but that doesn't replace the brain.

I already know which redheads have big ones on my local system and where to find them no matter what the file is named and the computer can't find all of the redheads with big ones on the web unless they all have the proper metadata attached to them. Some file out there somewhere named 1038754875747.jpg is just as anonymous to the computer as it is to you.

KFG

Re:Performance?

[EddWo]

Why don't people get it? It isn't about entering arbitrary keywords as file properties, it's about forming relationships between items. You shedule a meeting, you create a new meeting item and link in the attendee items, location items etc. You take some notes at the meeting, you link the notes to the meeting item. You link the meeting item to the project you are involved in.
Now you can locate the notes by filetype, time/date, project, meeting, people, location, without entering any text by hand, just linking one or two critical bits of information per new item.

"Implementing in GNOME"

[kosmosik]

Such thing should be implemented at kernel level to be transparent for *any* aplication. Without this it will just lead to a mess (like 4 different implementations) and some apps working with it and most not. As f.e. you can browse SMB network with Nautilus but when you actually try to open a file (from SMB via Nautilus) in OpenOffice.org you will get a info that viewer does not support this method... It must be a standard system routine not another level between system and GUI.

Re:"Implementing in GNOME"

[slimyrubber]

Actually database storage should be implemented in the filesystem rather then kernel or window/desktop managers. That would make much more sense and theoratically, it will be faster.

Just my uneducated opinion.

Re:"Implementing in GNOME"

[Henk Poley]

No, not in kernel. It should be done as a deamon, like he did.

Re:"Implementing in GNOME"

[justsomebody]

Yeah right. And whole world will use this daemon.

Problem with this logic is that not everybody is gonna use DBFS. For example: Some people would like to use Reiser4.

Proper thing would be dummy kernel (or some higher VFS, but making whole thing independant of wheter graphic mode is present or not) API for this kind of file access. If accessed FS is not DB related then it should convert standard functions (implementing some Metadata index on basic FS). If accessed system is Database FS then it should go trough it's native layer.

Who says that

MOVIES="*.avi *.mpg *.qt"
for ftype in $MOVIES; do
ls /mnt/volume/Personal/$ftype

aka.

select from fs where (path = '/mnt/volume/Personal/*') and ((path = '*.avi') or (path = '*.mpg') or (path = '*.qt'))

is different than

select from fs where ((type='movie') and (location='Personal'))

Second option could pose other options. Like searching by actors and directors, MP3 info, Office tags etc. But all open and save routines could be done trough and with Metadata. While Metadata and data would be connected.

Let's say I copy MP3 files from CD. CD is not DBFS. All my tags go down the drain. Hope you don't expect I will copy all files trough this daemon or in graphic mode at all. Just as on plain system as on DBFS.

Off course, it would help if VFS layer could detect MIME and act accordingly. Example: Old ISO9660 burned CD (or even better ssh mounted drive) with MP3. I copy these files to my computer remotely from terminal console on some other computer, computer being copied to, resolves ID3 and updates Metadata from ID3 tag index on fly. Without having some cron job.
Same thing goes for my Office files. All files have it's creator and it's description. Why wouldn't go in Metadata index when file is received and saved from e-mail. If this problem is not solved before implementation then all you can expect are holes in your Metadata with a lot of non-indexed data.

Well, that example is nothing new. Reiser4 already does that with plugins. The question here is:

Will everybody use Reiser4? NO
Will everybody write metadata plugin for Reiser4? NO
Will Gnome or KDE support Reiser4 directly? NO

Would this have better chance when Universal file access would be defined independent from FS and independant from Graphic mode? YES

stubborn

[mn3m05yn3]

Article doesn't address whether or not we can turn DBFS off and use the more traditional hierarchical method of file placement. Will we be dragged into this kicking and screaming?

Re:stubborn

[donkeyboy]

Most of these DB File System implementations overlay the existing file system. The standard file access methods are still there. The DB functionality is implemented as extensions.

The DB tables are then implemented as special files that coexist nicely with the existing directory files.

Keep in mind that if a "new" file system breaks heirarchial file access is also breaks every app in existance.

Version control would be nice as well

[zero-one]

I have always thought that version control (file histories, branching and atomic changes) would be nice to have at the file system level. Instead of storing myessay-firstdraft.doc, myessay-seconddraft.doc, myessay-final.doc, the file system should do the work. Then if I want to make a bunch of changes (perhaps I want to try a new page layout), I should be able to commit them as one atomic change (or throw them all away if I change my mind). Then, when I want to make a set of documents with US spelling, I should be able to branch the whole lot (using no disk space) and make the small changes from UK spelling while still being able to integrate other changes I have made.

Re:Version control would be nice as well

[ultrabot]

I have always thought that version control (file histories, branching and atomic changes) would be nice to have at the file system level.

Sounds like a job for an SVN plugin for Reiser4 file system. Anyone doing one already?

Re:Version control would be nice as well

[Anonymous Coward]

There are two filesystems I know of, off the top of my head, which support versioned files. The first is VMS and the second is ISO9660. I suspect the VMS file versioning probably came from TOPS so you could be right there.

Yes, ISO9660 really does support versioning. You can create files and add a version number using the ; seperator E.g. FOO.TXT;1 and FOO.TXT;2 etc. A compliant ISO9660 can either show you all of the available versions if your OS can cope with it or just chooses the latest version (E.g. for systems such as MS-DOS). I'm not totally certain how versions are handled if you use RockRidge or Joilet extensions; the original ISO Level 1 names are still there but you'll only see the extended filename if you mount a RockRidge disc with a RockRidge ISO9660 driver, so you probably won't see the different versions. It's been a while since I fiddled with an ISO9660 driver to be honest.

Re:Version control would be nice as well

[david duncan scott]

Close. That's a VMS feature.

Disadvantages

[BHearsum]

How much permforance overhead will this cause? The 'Desktop Environments' already eat a lot of RAM and CPU.
How much disk space will you lose over this? All the metadata has to be stored somewhere, and just glancing over the link I read something about a versioning system, which will definently take up quite a bit of space. Will a 20gb hard drive become 15gb with DBFS?

Re:Disadvantages

[aodl]

While performance is something that should always be kept in mind, we are a long way away from the days of the original Macintosh where a desk accessory had to weigh in at 600 bytes in order to make the cut and fit into both memory and on a floppy disk. As current desktop machines outperform the high end servers of a few years ago, it would be nice to put a lot of that muscle to use in improving the user experience. I'm not excusing bloated and slow code here, but we don't really need to be counting bytes.

In any case, database based operating systems have been around for decades, from OS/400 to the BeOS. Many BeOS users claimed it was hands down faster than any other shipping OS at the time, and it featured a journaling, database-styled file system. One of the primary developers of that file system is now working at Apple on Mac OS X 10.4's spotlight functionality.

The thing is - as our desktop storage continues to grow at the pace that it does, and as we curiously find ways to fill it up, new ways of looking at and finding the information we store are going to be needed.

DBFS, Gnome Storage, Apple's Spotlight, and WinFS, all take different routes to get there. It's worth looking at all and what they offer and where they differ. WinFS, is a new storage layer that combines file system resources with more structured data in a Relational/XML hybrid system, with the aim (from what I gather) of turning the file system into a global "soup" of data. That sort of soup can be seen in office suites or PDA style applications, and in older Operating Systems like the Newton OS, where everything is a shared and available resource that is stored and available through common structures. Spotlight, on the other hand, combines file system searches and indexes (think 'locate') with full content indexes and a metadata index, which uses 'importers' to parse out other file formats. Spotlight is not a new file system, but an indexing system that acts on files in the file system. From what I remember of Gnome Storage, it is similar, using the VFS layer and Postgres triggers and callbacks, along with plug-ins, to parse and extract relevant metadata and contents out of files. DBFS looks to be like WinFS in that it purely wants to be a new kind of information store. I don't know which style will win out. My theory is that technologies like Spotlight will eventually evolve into a new kind of storage system, while remaining familiar and file based for todays users and developers. But this is an idea whose time has more than come. It's something that's been promised for the desktop for at least a decade, and has been shown to work, albeit in targeted OS's (the Newton) or ones that never achieved mass market penetration (BeOS).

So I think that performance concerns aren't that big of a concern, so long as (like all development) there are good people working on the solution.

Re:File system in KDE...

[mn3m05yn3]

Doesn't seem to be true. Article indicates the DBFS will run on top of other filesystems. if you don't use KDE, you don't use DBFS... ext2/3 etc is still there. The real question is whether you can use KDE without using DBFS...

Reiserfs, storage and why do you want this?

[ACK!!]

Perhaps I am more organized than most but I already categorize my files and such in the hierachal file structure.

Isn't Rieserfs planning to do this on the kernel level?

Where does that leave other fs choices and storage and other idea dbfs?

I see more and more people saying look what neat things you can do with these tools.

But really why do you personally want something like this?

Curious to see the response is all.

Re:Reiserfs, storage and why do you want this?

[BenjyD]

The problems with the hierachical system are:

- maintenance overhead on part of the user to create hierachy and maintain it. Every time you save a file you have to think "where do I put this?"
- Finding files can be hard. Is that letter about the planning application in Documents/Letters or Documents/Planning App?
- keeping files in two or more places at once is hard (as in the previous example). You can use softlinks, but that's hardly ideal and doesn't survive moving things around.

Basically, the current file system imposes a significant overhead. Most power users have restructured the way they work and use a computer in order to minimise that overhead without really noticing. It's just become one of those things you have to do, like remembering to save documents.

Why not shift the burden of organising the files onto the enormously powerful computer, rather than take up valuable human mental resources.

Re:Reiserfs, storage and why do you want this?

[Aeiri]

- Finding files can be hard. Is that letter about the planning application in Documents/Letters or Documents/Planning App?

find ~/Documents | grep "planning application"

You've just taken a giant step into learning how to search for files in UNIX! In my next tutorial, I'll teach you the power of the "locate" command.

Re:Reiserfs, storage and why do you want this?

[TheRaven64]

I find that a spacial interface to a hierarchical storage layout makes it very easy when I want to find my files. This kind of thing is more useful when trying to find files you didn't create / save.

Re:Reiserfs, storage and why do you want this?

[ctr2sprt]

OK, let's consider an example other than documents. Joe is a big music fan with a couple hundred CDs. He likes having instant access to any one song when he wants it and has a lot of hard drive space, so he's MP3-ized (or ogg-ized or flac-ized or whatever) his entire collection, plus all the MP3s he's acquired via other means.

Joe is pretty good about organization, so he names every MP3 properly with the group, album, and track names, plus the track number. (Something like The Beatles/White Album/01-Back in the USSR.mp3.) This way if he knows, for example, the track name but not what album it's on, he can find it pretty quickly using a method like yours.

The problem is if he wants to do something like put all the country music he has, for example, in a playlist. How does he do that? It can be done, certainly, but if he has a collection with several thousand MP3s it's so tedious and difficult as to be effectively impossible. What if he wants to listen to 60s rock? What if he wants to find a particular song he has, but all he knows is it's between 3 and 5 minutes long, came out between 2002 and 2003, and is probably categorized as either "pop" or "alternative?" What if he just wants a list of all the songs he never listens to because he's sick of what he's been playing lately? Or maybe he needs to free up disk space and wants to find out what he'll miss the least.

All these things are impossible to do in an efficient and timely manner using our current system. He can certainly use a command-line ID3 tagger to strip out the things he cares about, something like

find mp3 -type f -print0 | xargs -0 id3tag -l | grep 'Genre: Country'

but that's painfully slow: a second or two per file means a big connection will take 15 minutes or longer to scan, and if you typed "Gerne" by mistake you have to do it all over.

Now if you had a filesystem-like object which could be smart and store ID3 metadata in the filesystem, then it would be much faster: the main overhead to using the find/xargs/id3tag/grep approach described above comes from having to seek through the MP3 file to get at the metadata. The reason this needs to be a "core OS component," perhaps even part of the kernel, is that MP3 tags can change at whim and the filesystem needs to know about it or its metadata can get out of sync. It's possible, but impractical, to update this on a periodic basis, like the locatedb; it makes much more sense to have the kernel inform some plugin "Hey, this file just changed, do you care?" And if the plugin does care, it can look at the changes, see if it's affected, and possibly update the metadata to match. It could also go the other way, where Joe updates the filesystem metadata and the OS knows to update the MP3's ID3 tags too.

Comments by the Author

[data1]

The author is asking for help to move the project to Gnome.

Quote:There is of-course the hard choice of platform. I choose KDE? because I am familiar with QT a bit, and because it is inherently object-oriented, being C++ and all. But in my mind GNOME? is much closer to how I would like a desktop system to function. So I would like to go for the GNOME option. I leave KDE developers to do what they want with this, and I am offering them support. But I would like to focus my efforts on GNOME and implementing the above in GNOME.

Any volunteers?
In the first place we will need developers. Would you like to join, send me an email (o.gorter@student.utwente.nl) with DBFS and JOIN somewhere in the subject. If you are not a developer, but still would like to help, please revisit this page in a few weeks. There will probably be a community website by then somewhere.

What happen to the OODB?

[jeanicinq]

The database file system originated from the ideas of an object-orientated database. Keywords and references are all part of the orientation objects of the database to index to files or other objects. It does away with the traditional hierarchal view, being rooted at some place. The OODB does not need to be rooted as it is more like a web. The DBFS seems to try to implement part of the concept of the OODB. Good. There are many more features an OODBFS can offer: dynamic organization, classification, and mutliple "skeletal" views to name a few. I hope that this DBFS will give a taste of what an OODBFS offers.

Sigh, Andrew Morton seemed to be right...

[greppling]

...when he said on LKML, slightly paraphrased: "The only reason I see to put filesystem semantic enhancements into the kernel, is that it would be socially hard to get people to agree an a single userspace library."

(In the course of the heated discussion about Reiser4.)

SharePoint anyone?

[Petronius]

While everybody is busy making fun of WinFS, Microsoft is very quietly and successfully letting their customers install SharePoint sites all over the place.
As usual, Xerox came up with the concept years ago (DocuShare). Sigh.

Re:SharePoint anyone?

[Anonymous Coward]

It feels like sharing files in the manner that SharePoint does is an afterthought. First, they built a web portal, then they decided to add that feature in, along with a few ActiveX controls. More important to SharePoint was the web portal component. Where I work, everyone has a SharePoint site for a default webpage. This allows for simple announcements and memos without choking down the email system. It's also fairly unobtrusive, which allows users to read said memos if/when they like, rather than forcing it into their inbox where it would inevitably get deleted.

This is not a file system

[MobyDisk]

Maybe we could call it a "filing" system since it indexes files that are on another file system. Really, a file system IS a database, not an add-on that indexes files. Still, perhaps this is a better approach than trying to redo all the file-system internals. Although to be truly useful, this needs to be an API that is GUI-independent, with GUI-bindings as needed.

Stop letting goof ideas

[DarkOx]

Stop letting good ideas be victimized by the M$ marketing FUD machine. Someone said "database filesystem" That was a good idea. M$ can along and said gee lets steal that idea. Hey there is no existing implementation to copy how do we do it. The answer is they did not do it. They put a database to keep rather mundane information on top of NTFS(a real filesystem) and called it WINFS(NTFS and a almost completely unrealated database, not a file system). Keeping data on the relations ships between files is a nice idea. Putting it in user space is dumb. Its overhead. Look at most of the example he gave "find a word document I worked on last month". All that info is already in the filesystem. A filesystem really is already a database in the strictest since. It stores whats on which inode assoicated with how many blocks which you could think of as attributes. It also sores attibutes like permissions and dates. Why not just put some more attributes into it like subject and relatedtopicID . If you did that and then added the ability to maintain some other tables where you could put extended descriptions and stuff, and built up the query engine to be able to efficently solve queries users will likely ask then you'd have what your really looking for. Addionally you would lose the overhead to a degree because you'd be storing informaiton once instead of in the FS and in the database.

This looks like BLINKX plus more

[lcsjk]

I already use blinkx, beta, from http://www.blinkx.com/, to automatically search my files along with internet keywords. It doesn't have the search by date or extension and is not configurable to my liking, but it seems to do a good job of finding things I have misplaced. Integrated with the author's system, this could make a great search system.

Normally I file things in a hierarchial method by year and month and by project name (2004file/9sep/) or (2004file/workfile/projectname), but still I lose track now and then and need keywords. Change the "slash" slant to fit your OS.

AVFS - A Virtual File System

[mooobo]

Have a look at this for a userspace filesystem http://www.inf.bme.hu/~mszeredi/avfs/

There are a debian package called fuse-source and fuse-utils. I.e. use all these nice kio plugins on filesystem level.

Just add

deb http://www.kalyxo.org/debian/ unstable main
deb-src http://www.kalyxo.org/debian/ unstable main

to your /etc/apt/sources.list

apt-get update
apt-get fuse-source fuse-utils

Actually, I didn't test that yet. Someone else?

What About Code Bloat?

[reporter]

These days, operating systems like both Linux and Windows XP have too many bells and whistles that I simply do not need. Unfortunately, these bells and whistles drastically increase the amount of space that I need on my hard drive.

Adding a database layer to Gnome sounds like using another 300 megabytes of storage on my hard drive. I simply do not need the database.

If the FSF/GNU folks really want to do something revolutionary, they should fork Linux+Gnome into 2 distinct paths: minimalist and maximalist. The maximalist is what we have now. The minimalist is a minimally featured Linus+Gnome distribution. It is the bare minimum in functionality that we need to have a decent operating system and desktop.

Into this minimalist installation, I will then add the applications (e.g. MatLab) that I use daily.

Re:What About Code Bloat?

[M. Baranczak]

You don't need to do anything to Gnome. There are plenty of existing Linux desktop environments that sound like they'd work for you. Take a look at xfce. If you want it even more stripped down, then try BlackBox, IceWM or WindowMaker. Most distros include all of the above.

higher level of abstractions...

[3seas]

Keeping the traditional file system is only logical as this database sort of file access is in fact a higher level of abstraction.

Considering there are numerious project of such higher level of file access abstraction going on, it does become a secondary choice for the user if they want to use one of these higher abstraction level file access systems.

To remove the traditional file system altogether would be a mistake, as then it could become a system of babel or keywords.... "what was I thinking when I created that keyword and lets not even get into what crazy joe was thining when he came up with his keywords...

But hey, given how MS based developers would create some obscure dll name and place it in some obscure location in order to copy protect .....

higher level abstractions are useful only to the point that you can, if need be, drop down in abstraction level to get your bearings as to where you are. If you cannot touch physical reality then how do you know you are not floating around aimlessly?

being out of touch with physical reality can evenm be very dangerious and hard to correct.

standard filesystems are NOT databases

[lkcl]

standard filesystems only have ONE index - a hierarchical one that contains a certain amount of real-time-updated indexing (such as the timestamps on a directory)

but it is NOT a relational database: you CANNOT easily create or use an alternative index to your files.

that's what all the fuss is about.

some people mentioned here that they already organise their files. great. fantastic.

HOW LONG DID IT TAKE YOU?

and how long would it take to reorganise?

with a relational database, all your indexes are updated AUTOMATICALLY.

therefore, doing searches on a relational database filesystem (find me all music files with dates between last week and last month: SELECT * from files WHERE files.type = "music" and files.date NOW() - 7days

you _can't_ do that sort of thing on a traditional filesystem.

sure, you can emulate it by creating symbolic links all over the place, but what happens when a file is deleted or moved? you need to manage / relocate the symbolic links...

nah.

databases.

fantastic idea.

now can we have them as a kernel module, pleeease?

Re:standard filesystems are NOT databases

[mgkimsal2]

therefore, doing searches on a relational database filesystem (find me all music files with dates between last week and last month: SELECT * from files WHERE files.type = "music" and files.date NOW() - 7days

you _can't_ do that sort of thing on a traditional filesystem.


Argh! Don't say things like that - someone will throw down a shell script which WILL do it (probably in one line) combining find, file, grep, perl/python and some other crap. Which, while it may work for some, entirely misses the real point, is that there's no *easy* way to do this. We've got GUI bindings for opening/saving/editing files, but nothing for harnessing the power of searching. Don't say it can't be done (because it can somehow) just say that it can't be done easily, which is the bigger point. :)

Read the article, read some history

[eschasi]

To quote from the article (which most folks have not read, as usual):

The DBFS does not actually store files, it holds references to files on the underlying hierarchy based file system.

That line alone should answer many of the questions re backup, speed of FS performance, etc.

At a deeper technical level, nany of the questions asked here have historical answers or clues in The Design and Implementation of the Inversions File System. The abstract reads:

This paper describes the design, implementation, and performance of the Inversion file system. Inversion provides a rich set of services to file system users, and manages a large tertiary data store. Inversion is built on top of the POSTGRES database system, and takes advantage of low-level DBMS services to provide transaction protection, fine-grained time travel, and fast crash recovery for user files and file system metadata. Inversion gets between 30% and 80% of the throughput of ULTRIX NFS backed by a non-volatile RAM cache. In addition, Inversion allows users to provide code for execution directly in the file system manager, yielding performance as much as seven times better than that of ULTRIX NFS.

Note that this paper was published in early 1993. Many of the issues it addresses are relevant to DBFS, and many of DBFS's advantages are foreseen by that paper. IMHO DBFS has chosen a direction that should have better performance than inversion, not to mention lower risk and easier failure recovery.

Inversion was built on POSTGRES, which makes one wonder what happened to the source.

blah blah blah

[Anonymous Coward]

Sorry, but I have to say it, you are an idiot!
Did you even care to RTFA?
This has nothing to do with the gnome or kde devs. Some developer invested his time to come up with something he thought was useful and all you can do is complain?

And if you look at the project, it is something completley different then a plugin for the admittedly great ReiserFS4 and it is here and usable right now.

So friggin stop your stupid whinig.

And to the mods who modded parent interesting ...

Btw., why don't you whine to the Reiser people that they should stop developing now? It would be as justified as your whining aboutt this project.

I can't see the gain.

[The Mighty Git]

A db fs with rich searching of metadata requires the orderly and accurate entry of said metadata.
You can't get organisation out of nothing - you are just asking people to be organised in a slightly different manner.

An organised person can already work effectively in a filesystem with current tools. The fact that they are organised is the key.

A disorganised person is not helped as their metadata will be erratic or absent. In fact might they now have the capability to be even more disorganised?

As I see it this is not solving an organisational problem as much as shifting the interface to the problem. I do not believe it to be either an easier or better way to organise personal data. Conversely I do not believe it to be inherently worse either. It's just different.

Where's the gain?

Apple's Spotlight

[DuckWing]

For those of you that have not yet looked at the Mac OS X (Tiger) preview and WWDC web cast, the new spotlight technology built into the next version of Mac OS X looks very much like a fully integrated database file system. And it's incredibly fast. Go check it out!. Note: QuickTime required. Mplayer may work for us Linux heads but I haven't tried it.

Any innovation is good

[Stevyn]

People can offer their opinions for or against this, but I think that any innovation benefits linux. I've read about WinFS and it sounds like a good idea, but who knows when it will be ready. If people working in their spare time can get something like this working in linux before Microsoft can get it out, I think that would just be another reason to trust the open source model of developing code and squash Ballmer's FUD.

I don't have too much trouble using a hierarchy file system. I keep my stuff pretty organized, but computers are supposed to save time, not create more problems. If this database can do a good job, I'll give it a shot.

please! not KDE vs. GNOME again

[whovian]

This basically seems like giving find the capability to do file and then storing the results with locate. And add some time stamping, sorting capabilities and GUI.

Since all but the GUI are basic commands, it would seem sensible to have an underlying library with hooks for use by your choice of desktop manager.

Re:i don't have time to reinvent the wheel today

[uncommonlygood]

why don't the nice KDE people and the nice Gnome people work on developing a library that sits on top of [reiser 4] and then we can stop all the stupid name calling and use the right tool for the right job

I wouldn't normally jump in except this is modded +4, even though the poster doesn't appear to have read the article.

The article doesn't talk about an actual "file system" (it admits its a bit of a misnomer itself), merely a way of referencing files with "keywords" in database that, according to the author, will make it easier to find your files. The author has written a file browser and file selector dialogs for KDE that use this, then goes on to say that he's planning to make a GNOME implementation soon, but nowhere does anyone start any name calling.

Since this article essentially discusses a GUI and not a file system (which could be implemented at kernel level), it would be a little silly to use the same tool for GNOME and KDE, since they have different look-and-feel. That said, a common library of functions needed by both the GNOME and KDE versions would obviously be sensible, and would speed up porting to other desktops.

Furthermore, this idea claims to work on top of existing heirarchical file systems (removing the idea of a hierarchical FS completely would involve a major restructuring of the whole operating system and everything that works on it), so just saying that it should use reiser 4, I suppose because its just l33t, is a little redundant. This should work on top of just about any FS, ext2, reiser, or even an NFS mount.

Not Unix

[flossie]

the DBFS does not store system files: No shared libraries, no font files or others like that. These are not documents, not files you look up at a day to day basis, and have no place in a file system.

Whether or not you look at system files every day probably depends on what you are doing with your machine and what you consider "system files" to be. Moreover, this idea would seem to go entirely against the whole UNIX "everything is a file" philosophy which is supposed to be one of the great strengths of UNIX.

SVN + DBFS + 10TB Nano Hoozie

[KrackHouse]

I'm using Subversion for a project and the idea of Atomic Commits seems like an obvious direction for file systems. If that other slashdot story is correct, storage becomes less of an issue and it would be possible to roll back the system to any point in time or to only roll back one file if need be. Now throw an intuitive way to navigate files on top of that and you've got a sure winner.

In the grand scheme of things, only a very small handful of us on earth are aware of Linux or even know what an Operating System is for that matter. File systems seem to be the big stumbling block for new users. Anything that can make computers and therefore access to information easier for the coming waves of new computer users (maybe billions of people?) will be a good thing. Even if the "bloat" slows down the system by 10%.

I hate to preach but that old quote comes to mind "With great power comes great responsibility". I don't think most of the people working on the OS that will soon dominate in developing nations (that's Linux) are aware of the harm they can do by slowing down Linux development with petty personal disputes. Like it or not, Linux is no longer just an edgy hacker tool. It has the potential to change the lives of Billions of people.

Backups

[Thu25245]

"How would a user be able to group (manually) related files together, so that the whole bunch can be backed up later, without having to search for all seeminly related (or unrelated) keywords to trace all hitherto-unrelated documents? "

Search for: Documents
Modified since: last backup

I don't know about the other implementations but Searchlight and WinFS are implemented atop the existing filesystem. (The FS in WinFS supposedly stands for "Future Storage" and not "File System") Sort of like how Google is implemented "on top" of the regular hypertext-linked internet.

This is not a new idea.....

[Cnik70]

Take a look at an AS/400 (iSeries). They've been around for more than 10 years. And before that you had the System 38 and System 36... and so on..... Why is this some big revelation now?

Exec 8

[jefu]

Univac's Exec 8 (I think), its been quite a while since I used it, had versioning integrated into the files themselves - at least for some files. That is, they marked changes to the files as part of the file content up to 5 (I think) levels deep.

This did mean that you had to use the right tools to get into the files or you had to cope with the changes in programs that worked on them.

VMS also had file revision numbers on files as a couple of posters have noted.

Both of these were nice in some ways, but relatively difficult to deal with in other ways. By comparison unix is straight-jacketed but easy to use.

Set Theory to the rescue

[Tablizer]

What about backups though? How would a user be able to group (manually) related files together, so that the whole bunch can be backed up later, without having to search for all seeminly related (or unrelated) keywords to trace all hitherto-unrelated documents?

Defining "group" is one of the problems with hierarchical systems. What is the group you want is dependent on your needs at the moment. Relational makes it easier to create ad-hoc groups.

For example, do you want to backup by file age, file type, and/or by topic? And in the real world topics naturally overlap. Set theory has an easier time with this because it is meant to deal with overlaps; but hierarchies get messy beyond about 3 orthogonal factors. Relational is closer to set theory than trees are.

A drawback of sets is that most people are not familiar with non-tree arrangements. There will probably be a "training hump".

More about sets versus trees.

Little research won't hurt

[zdzichu]

Before inventing something you should check if no one did this earlier. Because there you have GNOME Storage. Don't be fooled by screenshots there. Storage isn't only cool search facility with native language parser ("computer, find me all porn I've downloaded yesterday" anyone?).
Storage is, suprisingly, method to store files decomposed to contents. The great searching ability is a side effect.

Imagine collaborating in of group of people over one document. Every one got some paragraphs to edit. With Storage, everyone can edit this document in the same time, seeing other's changes as letters are typed. Store version history and you have revision control. Throw in network transparency (you go to other department, connect laptop and automagically you can work on those department files) with OpenTalk (Zeroconf/Rendezvous) and you got best idea since hierarhical directories.

Be sure to read whitepaper about Storage available on mentioned site. Also check for Storage related entries in Seth's blog (Seth is one architect of GNOME Storage). Now if only KDE people work on compatibility with Storage, freenix desktop would rule the world.

BTW, KDE, don't miss chance of integration! KDE is planning to introduce google-like search in desktop. Don't reinvent wheel! Beagle is here, working. Just integrate Beagle with KDE desktop and we are set.

Jeez, when will these people learn?

[Grendel Drago]

Joel on Software said it best:

For example, WinFS, advertised as a way to make searching work by making the file system be a relational database, ignores the fact that the real way to make searching work is by making searching work. Don't make me type metadata for all my files that I can search using a query language. Just do me a favor and search the damned hard drive, quickly, for the string I typed, using full-text indexes and other technologies that were boring in 1973.

Why you don't want a DB-based FS

[0x0d0a]

I hate GNOME-VFS and KIOSlaves and all that stuff that people try to roll into a higher layer than they should. The idea of combining VFS functionality and a *desktop environment* is *stupid* and exactly the sort of fragmentation that hurts everyone. Want a userspace VFS layer (like, other than the existing transparent systems like LUFS and FUSE, which are much better if you can use them, because existing apps continue to work) Fine. Make one. Call it libvfs, and make KDE and GNOME bindings. But for the love of God, quit trying to roll filesystem functionality into desktop environments. It's ridiculous, and not where it belongs.

That being said, I'm still not a huge fan of this.

There are three main features that people seem to want with a DB-based FS:

* Transaction/views/high-level-DB functionality.

You don't want a FS that tries to do all this. DBMSes have worked for a long time to do this efficiently. If you want this, use a real DBMS, because it's going to smoke what someone tried to hack up into a filesystem.

* Automated index updates. Basically, locate but atomatically updated as changes occur.

Mac OS Classic had synchronous index updates as the filesystem ran. This makes the filesystem slow. It sucks for servers.

It's a much better idea to do asynchronous index updates, where the index approximates the filesystem quickly, but perhaps not right away -- you can do that without killing performance. Basically, you have the kernel notify a userspace daemon when changes occur, which rebuilds the part of the directory hierarchy (possibly waiting a while to see if it can use idle time). If a ton of changes occur to a small portion of the directory hierarchy, instead of trying to keep up with each change (say, a million changes to part of the directory hierarchy), the update daemon rolls all those queued up updates into a singled queued up full update of that entire portion of the directory hierarchy. It does a bit of extra indexing, but doesn't get backlogged.

This *could* be done under Linux, but there is one bit of kernel support that is missing -- currently, it is unusably inefficient for an app using Linux's existing dnotify() directory monitoring mechanism to deal with (a) changes to a directory containing many entries and (b) changes to directories anywhere on a filesystem (currently, dnotify() requires a FD for each directory to be monitored).

There *is* a enhanced dnotify patch out that would improve Linux's dnotify() mechanism to the point where people can write daemons with this, but Linus has not yet rolled it in. Once he does so, we can get the ball rolling on such daemons.

* Indexing of metadata.

People want to be able to search for their files using metadata. Again, this can just as easily be done using such a daemon. Existing apps continue to work, people can choose where to put something in a unique hirarchy so that they can reach it again, but files can also be addressed via metadata. If you want to provide, say, a tabbed file selector containing a tab for selecting files via metadata, that's quite feasible.

A major reasons why you don't want to provide a full DB-based interface is that you lose the existing hierarchical representation, and every app in the world stops working. You don't want people to just "save this file to a filesystem" and have to address it via metadata -- you want the user to give the thing some kind of unique identifier.