Slashdot Mirror
newdocms: Beyond the Hierarchical File System
Manuel Arriaga writes "After two years of hard work (and many scrapped versions), I have just released a (ugly, but working!) preview version of newdocms, a completely new document management system. newdocms isn't a file browser: it is a layer between the hierarchical file system (HFS) and the user, which provides a radically new way to store and retrieve documents. No longer will you browse complex directory trees or directly interact with the HFS; instead, you define any number of document attributes when saving a document and then query a database of those attributes when trying to retrieve it later on.
For the first time you have a true alternative to the hierarchical file system at the OS level. Through the modification of the KDE shared libraries, newdocms currently works with all KDE apps! (I am looking for volunteers to add support for GNOME and OpenOffice.org!) This is a testament to the power of free software: this sort of innovation could never happen if it weren't for the free software nature of the underlying systems."
I'm already using The Brain. It's *really* unique, and it works. It works very well. And, in addition to organizing files the way YOU want them organized, it also connects random thoughts, web sites, emails, etc. If you haven't seen it, check it out. It's pretty damn incredible.
What Microsoft suggested something like this, everyone went mental, and I got bitch slapped for saying I thought it was a good idea.
- Not confusing enough.
- No possibility of new patents.
- Lack of ability to lock users into your proprietary file system.
I didn't know HFS was broken.NetInfo connection failed for server 127.0.0.1/local
I have worked with many a user that has had problems with the concept of folders (directories). Perhaps those users can grasp this concept easier.
1. "Filesystem? I don't need no stinkin filesystem!" An ideal Palm-esque computing environment wouldn't have any filesystem. There simply isn't any reason for it. Why would you store addresses in an address file or a book report in a word file? Saving/Opening files should be transparent to the end user. Versioning should be built in, yet simple to understand. Forking files can be accomplished without copying a file. This is intuitively the simplist idea.
2. If you somehow *have* to think in terms of files, then your conclusion may be to use files. However, I don't see why anybody would come up with a hierachical file system, unless they were accomidating for hardware limitations. Placing files somewhere within a huge directory tree is just too darn complicated. Why should the same file not exist in multiple directories? Why should copies of a file exist? Everything, including advanced security policies (more advanced than what is currently possible) is available for a *keyword* driven filesystem.
I believe this is a step in the right direction and I can't wait until my favorite OS (not Linux) adopts a similar feature.
That's the whole reason for the program -- you shouldn't have to remember long, detailed folder structures and filenames in order to retrieve a file you were looking for.
I can't tell you how many times I've had to help users find some file, shortcut, document or spreadsheet that they've "lost" because they forgot the correct path. But they do remember it involved a loan, or it involved a party announcement, or something similar. I swear, just the other day I spent an hour waiting on another employee to get off the phone so I could find a folder shortcut another employee had lost. She wasn't sure what folder the shortcut referred to, but she knew it contained documents of a certain type.
Do you see a pattern here? To me, this sounds just like what Microsoft is trying to do with Longhorn, and potentially Office 11. People are tired of searching and hunting through folders and heirarchies full of oddly named files and temp folders that can confuse Joe User.
This is awesome software and definitely a step forward. It might not change the geek community, but it will certainly help out system admins of the world. While your method still works (and hopefully, in the future, these two systems should work hand-in-hand, but that's another project I suppose), this is a damn fine alternative.
I agree. Basically the only way this is different from your HFS is that it encapsulates the meta-data (that is currently in the path name) differently. I'm not sure that's any better or worse. In fact, I myself like to be able to see at a glance what all the categories of documents that I have are which is quite easy with HFS, but doesn't sound so easy here. Perhaps that's more because this is a new idea and not mature yet.
Everyone seems hot to SQL the file system, and while I think that will be the way of the future, I don't think that there is a clear view of how that works from the user's perspective yet. Remember that this is a rather large paradigm shift from what everyone is used to. It's going to take a while for this to mature to the point that Joe User is going to be able to hack it. I mean, I looked at the Save As dialog on that page, and while it looks cool it also looks counter-intuitive to me and I'm a developer! How much more will a user get confused?
All in all we're going in the right direction, but by no means are we anywhere near the goal yet.
Ben
Exactly. In fact, these hierarchies do not make sense to anyone, encountering them for the first time. There's nothing user friendly about them at all, really. They aren't even alphabetically sorted, which is the least you usually expect from a file cabinet. It's just the simplest way of doing things and it seems logical to you, because you haven't worked with any other kind of file system since you're first computer experience. Admittedly, a keyword driven system would not give you a shorter syntax. But administering a system using thousands of levels of subdirectories would not do that, either. Imagine a database driven file system, combined with near-perfect speech-recognition software. Suddenly the additional keywords required do not matter so much, and the advantages of a system like this could really become obvious.
This is exactly what I have been wanting for almost a decade now.
..etc as well as those that are simply wallpapers and photos). More importantly, if you see a good bump texture for a certain surface, describe it as such without changing the filename.
Some uses I imagine
- Create music playlists on the fly (MoodLogic doesn't count)
- Categorize work files (Across the whole partition, find images that serves as bumps, HDRI
- Install Windows and service packs first, mark files as "windows native". Then install apps. Some OS glitch, you need to reinstall ? Backup all files with directory structure which don't have "windows native" tag alongwith c:\program files and registry. Reinstall windows, restore the backed up files. Voila, no app installations required.
This looks a lot like something I've used in the past - FileNET Content Management Services. FileNET lets you create meta-data for each document you save, as well as a complete version history and check-in/check-out for each document if you want to. It also allows for hierarchical storage of files as well as using the meta-data so you can still categorize things by folder if you want, but still query documents by any of the indexes that you have built. It will even add a full-text search across everything in the library if you want, and it has no problems indexing most standard formats including Word and PDF files.
The system I have been dreaming of for a while would be far more graphical (had a quick look at thebrain.com, it's still text with a few lines as far as I can see).
My dream system would enable you to specify file attributes such as size, path(s), name, type etc, as well as regex greps on the content, and then plot the filing system in 3D space, through which you could move with a joystick. You would be able to assign attributes to graphical features, eg make scripts cuboid, text files spherical, bigger files bigger on a logarithmic scale and so on. Related files would appear like solar systems, and by changing the importance of the file attributes you could change the way the files grouped.
Probably not what you'd want to use every day, but I'm sure I'd find a few mislaid files with such a system.
Virtually serving coffee
Hierarchical file systems are as close to intuitive as you get. Everything you do in the real world, as pertains to dealing with information, mimics a hierarchical file system. Your chilton manuals are in the garage, your cookbooks and recipe boxes are in the kitchen or dining room, your computer books are by your computer. You don't look in the computer manual for how to change your oil. When you are trying to bake a cake, you don't walk out into the garage for inspiration. Having information organized into different places, and then having those places subdivided into different boxes is intuitive, and is how most organized people think.
v able\Yesterday\Tomorrow\A WeekAgoToday might be confusing, but the filesystem paradigm isn't.
1. (a) "We don't need no stinking filesystem." The ideal palmesque OS would have the same idea just demonstrated differently. You aren't going to open up your notepad to see an address. The address file is in the address program (directory). The schedule file is in the calendar program(directory). The programs you use to open the files become your folders.
1. (b) "Saving/Opening files should be transparent" The only people that would think like this in the real world have been living with someone that picks up after them all the time. When you are working on some (paper and pencil) project, and just stand up and walk away, do you exepect it to be available at the office tomorrow? When you start working on several projects in succession on your desk, and have reams of loose paper, can you easily bore your way back down. No, reasonable, organized people pick up the porject they are working on, file it away in the file cabinet/brief case/wherever it is supposed to go. There are logical beginnings and endings to your working on a project that only you can decide on. A spreadsheet, for example, do you want it to save every time you make a change... No, by their design, you would normally set up all your formulas, save that, and then every day/month/year open up the spreadsheet, plug the numbers, get the results, and save the specific results to a different file, or just look at the values produced. Not to mention, when you sit down at your desk in the morning, do you expect your desktop to know what project you want to work on? No, and you don't expect your computer to know what project you are working on either. Opening/Saving files shouldn't be and can't be transparent to the user.
I used to use a lot of floppies when growing up. I appropriated a lot of disks from other places. I used the "grab the black disk with the couple of remnant label pieces... no the other black disk... No, the one with the two small pieces of adhesive... Ooops, the one with the three pieces..." Now, I have to search all the disks everytime I want anything off of them, because I never labeled them. Saving things in well defined locations, for well defined tasks is reasonable, intuitive, and necesary task to saddle a user of any system/technology/information with.
2. I don't really need to address this point specifically, since the answer is inherent in the points above. The overly large filesystems are part of a whole system that the user doesn't really need to know about. That is why the "Desktop/..." paradigm of Windows came about, and is so useful. People working on your word processor have a reason to put the font files in one directory, the plugins in another, and the preferences in a third. The user couldn't care less. If you start the user in a directory tree just for them, then they won't be stuck in a huge file system, and can still work in a fashion that has made sense for litteraly thousands of years.
The filesystem paradigm has been around for a long time, again litterally thousands of years, because it works, it is easy, and it is how people think.
G:\Netowkrfilesystem\
Accounting\AccountsRecie
I've noticed about three main types of people in the world of open source: those who fix things, those who try to improve existings things (i.e., make it run faster, smaller, etc.) and those who like to tinker and make new stuff. This person seems to fit in the third category. As far as I can tell, this person is not so much trying to "fix" the file system, but to make a new and different version and/or approach to it. This may be a good thing. But if you don't like it, don't use it.
Life sucks, but death doesn't put out at all....
--Thomas J. Kopp
I think the tree model is ideal. What is not ideal is everything after the tree.
The file selection widget (FSW) is a core element of any high-level toolkit, and yet I've never seen one that provided any kind of utility that I need to make a filesystem work well in a GUI.
For starters, all FSWs should have memory, and they should understand what they're being used for. All of my graphics apps should "remember" where the last graphics app saved a file and default to that directory. Same goes for opening a file. Or office apps.
They should also have a history pull-down.
We also need a graphical abstraction for the filesystem (other than the MS-like horizontal tree) that customizes itself through use. If, for example, there are three directories that I load and save files to/from all the time, they should be the most obvious and accessible things in the tree.
Do these things, and graphical interaction with a filesystem makes sense.
As for a metadata filesystem, I think there's utility in it to some extent, but unless "rm" understands it, and it's easy to use from that level too, it's useless to anyone who really USES a UNIX(-like) system.
Actually I would LOVE to have everything accessable in a database somehow. I've been wondering about something using the userfs stuff. Not really mounting a mysql database as a usermode filesystem but having information from the system available that way.
I've found myself many times wishing I could just type "select location,filename from datastore where contents like %resume%"
SQL comes much more naturally to me than the find command does. I would love an easier way to index the contents of everyfile on my system by an arbitrary number of metadata and then have that accessable via a simple sql statement.
I remember Scott Hacker did something similar with BeFS and his webserver at somepoint but he's long gone as is BeOS.
Am I the only one that this makes sense to?
"Fighting the underpants gnomes since 1998!" "Bruce Schneier knows the state of schroedinger's cat"
"Filesystem? I don't need no stinkin filesystem!" An ideal Palm-esque computing environment wouldn't have any filesystem.
I've been thinking along these lines for a couple years now. Suppose a computing appliance, perhaps handheld, or not, didn't have a filesystem. How would you make use of the hard disk?
Suppose the software saves everything in memory resident database. No filesystem, and no disk. Everything stays in memory. But it is virtual memory. Every page in memory has a reserved backing store page on the disk. The disk partition for this OS is just a big swap area. The total size of your usable "memory" is the swap area, not RAM. Now powering off the device becomes very fast. And so does powering on. No more "booting up" nonsense. You press the "off" button, and almost instantly the device is off. No matter how much data you have, or if you were in the middle of a huge unsaved word processing document, the device instantly powers off and back on again. No artificial concept of "saving" a file -- just like PalmOS. You don't "save" anything. In fact, no artificial concept of computer files. (For flamers: I'm not outlining a fully fleshed out implemention here, just some rough ideas, think different.)
You can still move your stuff to other computers via. "syncing" or whatever you want to call it. It's just that higher level concepts are copied, uploaded, downloaded, e-mailed, etc. rather than a file (i.e. collection of untyped, unlabeled bytes). I may move my mp3's, and they are still categorized by artist, recording, date, label, etc., etc..
I've also been thinking that a filesystem such as NTFS or ReiserFS that allows attaching huge ammounts of metadata, or small amounts of metadata to any file would be important. For instance, my 4096x2048 digital photograph of the grand canyon (big file), should still be able to have a thumbnail (say about 128 KB) attached as metadata. Since the thumbnail is part of the "directory" information of the file, merely copying the file to another location retains all the metadata. (As opposed to Windows or KDE, where the thumbnail is another little hidden file somewhere near where the original file was stored.) Heck, I might want a graphic thumbnail metadata attached to an mp3 file. Of course, I suggest ReiserFS or NTFS because there should be no limit on the number of labeled metadata attachments, nor on their size. I should be able to attach metadata "Title":"Grand Canyon", "TYPE","TIFF", or "Audio Clipping":<5 MB of audio data> just as easily. When I move the file, the metadata moves with it -- but the metadata is not seen in the primary information flow -- i.e. sequence of bytes -- that make up the "file" data.
As much as I hate Microsoft, I expect that it is they who will do stuff like this first. Ideas such as I am discussing here will encounter lots of resistance from the old school. Just look at the resistance to the topic of this article in this discussion. (I remember when we had to had to organize and save our files ourselves, and we used stupid extensions like ".jpeg" as the only metadata, and it was uphill both ways.)
Drifting to a different topic, I wonder if true innovations at higher levels come from us geeks? We put up with the most abysmal user interfaces for so long that we are not even capable of recognizing a bad user interface. We are comfortable with what we've got. I frequently see the attitude: if I can learn this stuff, then you can too. If you can't get under the hood of your 1920's car and fix it when it frequently has minor troubles, then you shouldn't be driving. Where I'm going with this is that it may take talented people who are being paid to build next generation interfaces who follow someone else's vision who is not constrained by the present.
Just some opinions. I should quit rambling now.
The price of freedom is eternal litigation.
Case 1:
I'm your average home user, but even so I have about 100 documents I work on. However, I was smart enough to give them meaningful filenames and locations where it takes only a few seconds to find the file. Remembering attributes for each and every file would be a pain.
Case 2:
I'm a developer. I'm sorry, but I want file Y in F/O/O/BAR. I need something exact to describe where a file is at least. Anything else doesn't work.
Case 3:
I'm a mornon who doesn't give a flying-f*** about where I put my files, and I don't care what I name them. I already have documents in my C\:, C:\Windows/Temp, C:\sdf34\, and C:Documants. It takes me a couple minutes or two to find a file. What? I have to classify by keyword now? Who do you think I am? It needs to classify the files for me or I won't have any of it.
Case 4:
I'm a scientist/business man that deals with classifications on a day to day basis. I already have a database because I needed it to be efficient. If it was on the file system level, then it'd be pretty cool.
I can't think of any other positive cases where this product is useful. Thus, it's my bet that it'll be niche forever. Anybody got any other use cases that I'm obviously missing?
While I do think the work presented is a great idea, it seems to me that it's a lot of effort just to setup the system.
Thats pretty much the problem with meta-data based file systems. They're great for new projects, where you have a clean start and can actually add metadata to the files. The real problem is legacy data.
My home directory weighs in at just under five gigabytes, and has files dating back over ten years, and thats just the "personal stuff". My work partition has about eight gigabytes, which is mainly source code.
I'm really not going to be able to associate metadata with every individual file by hand. Until automatic tools come along that will data mine the file content and automaticlly do some minimal level of association.
On top of this a whole new generation of development tools needs to be written. At a very basic level you need a version of make that will build all C source files on the disk with associated meta data "Belonging to Project X, dated no later than last week".
When you think about it you'll realise that while as a concept its fairly powerful, we won't be switching to using this sort of thing soon. For the same reasons the semantic web and RDF are having problems getting adopted, metadata based file systems face real problems before people will start widly adpoting them...
Al.The Daily ACK - Eclectic posts by yet another hacker