Slashdot Mirror
newdocms: Beyond the Hierarchical File System
Manuel Arriaga writes "After two years of hard work (and many scrapped versions), I have just released a (ugly, but working!) preview version of newdocms, a completely new document management system. newdocms isn't a file browser: it is a layer between the hierarchical file system (HFS) and the user, which provides a radically new way to store and retrieve documents. No longer will you browse complex directory trees or directly interact with the HFS; instead, you define any number of document attributes when saving a document and then query a database of those attributes when trying to retrieve it later on.
For the first time you have a true alternative to the hierarchical file system at the OS level. Through the modification of the KDE shared libraries, newdocms currently works with all KDE apps! (I am looking for volunteers to add support for GNOME and OpenOffice.org!) This is a testament to the power of free software: this sort of innovation could never happen if it weren't for the free software nature of the underlying systems."
They work fine for me
It sounds basically like when you want to find a file, you go type in a few pieces of meta-data, and then hit "search". It's a way to do it, but it seems to me (and it's early, so bear with me) that it's easier for me to remember one piece of meta-data (i.e. the path to the file) than several (as it would seem with this setup, as you would have to present more than one piece of data to differentiate between different documents, let's say, created by the same author on the same day). Maybe I'm just used to a HFS, but I find it simple to open up a command prompt and type "pico /documents/foo/bar/fubar.txt".
Anyway, an interesting concept.
This is a testament to the power of free software: this sort of innovation could never happen if it weren't for the free software nature of the underlying systems."
How is an "ugly" beta version of an untested new file management system a testament to the power of free software? And why is this better than a hierarchical system? Hierarchies make sense to Joe user. Normalized databases (you did normalize it, right?) do not. And why on earth would I want to set all kinds of BS attributes on a file instead of just clciking File, Save As, and then hitting the little "My Documents" button in the window that pops up?
This is a testament to the power of free software: this sort of innovation could never happen if it weren't for the free software nature of the underlying systems.
This is completely untrue. There are lots of other options (like The Brain) that have been out for a while that have nothing to do with "free software". Hell, the fact that other proprietary systems (that are better, in my opinion) came out earlier shows that not only is "free software" irrelevant in this discussion, but it actually lags behind software driven by the profit model.
While I do think the work presented is a great idea, it seems to me that it's a lot of effort just to setup the system.
It would be ideal if the computer -- the thing that is supposed to make life easier -- did the clasification. Until that happens I cannot see myself even considering such a file access method.
-- bartman
My father is really going to understand that. Not a bad idea but the implementation appears to need work. Another interesting thing to note is that this is probably coded in C++ and is going to be a bitch once again to interface with scripting libraries. I love KDE but it is a difficult task to integrate other languages with.
Got Code?
Exactly. Users STILL have to create their own type of organization.
/documents contains documents. Duh.
/documents/work contains documents for work.
The problem is people don't want to be organized, so they look to technology to help them be lazy. Plus try explaining 'metadata' to someone. At least now you can use the file cabinet, drawers, folders, papers example to explain the layout to someone.
"I can't give you a brain, so I'll give you a diploma" - The Great Oz (blatently stolen sig)
Who came up with the idea of "folders" anyway? Not hierarchical trees, but the metaphor.
The biggest problem with folders is no one wants to be a file clerk and weed, sort, and file their docs. The act of socking away a doc should as mindless as possible, not because (all) users are mindless but because they have better things to do, and shouldn't spend a minute adding keywords to every doc they might never see again.
You know how it is -- you're searching and coming up with junk, and want to yell at the computer, do what I meant, not what I said! This would be one of my first pics for AI on a personal computer.
I agree folders doesn't cut it, though as a metaphor for explaining the tree it's not bad. The problem is the tree.
This idea was made by GNOME and now being inherited and implemented by KDE. Read here. And again please don't make Linux start to suck with that idea.
The Brain is an interface on top of your current FS. Things like this have been done going back to the days of the Leading Edge Word Processor (separate file to get around the 8.3 naming conventions).
I believe the point that this mad scientist was making was that he's completely replaced the FS with this new database-based one.
It's certainly not innovative, but it's something different I guess.
--- I wish I could hear the soundtrack to my life. That way I'd know when to duck.
Who needs this? As one poster put it, isnt the path the only real piece of meta data you need to find a file? Think about mom and dad. What do they want to know? "Where are the christmas pictures of the baby from last year?" "What happened to that email I sent my brother last week?" "Where's the latest copy of my resume?" and so on. Natural language aside, these are all metadata-type queries (mostly dealing with time and filetype data, both of which can be extracted without any additional effort by the user). I think that if such a system of searching files is ever perfected, we'd have a serious killer app on our hands. Isn't this part of what the "semantic web" is all about? Isn't it frustrating to everybody that even the best search engine in the world still can't understand "find me all books whose author is mark twain"? It seems like a logical progression to expect that. Just like most of us aren't searching the web for *pages*, but rather particular *informatin* on those pages, I think that Joe User doesn't care about looking for *files*, but rather the information contained in those files. Thus it's only reasonable that if you give a user a way of easily describing those files by something more than just a filepath, that it will then be easier to find the information later.
www.HearMySoulSpeak.com
or something very much like it a few years ago.
i used it and it works like a charm.
of course hierarchical file systems are easy to use, you can name folders after categories, and they are easy to backup.
Interfacer.
all your HFS are belong to us.
The whole desktop/file/filesystems may indeed be ripe for a new metaphor to help conceptualise them. When computers were the principal domain of workers, the idea of a desktop with files and folders allowed them to grasp alien concepts.
But computers are becoming ubiquitous, pervasive. Perhaps a new metaphor could be found. An example could be objects in rooms. Think of different folders as different rooms - all files (or rather, all streams) are objects in those rooms. Navigation between rooms is possible through doors.
Of course, as others have pointed out, the HFS ain't broken, so why fix it? (Answer: why not? PC cases aren't broken, but we still have case-modders, don't we?)
That was done pre-UNIX with PICK. The whole O/S was a database.
Microsoft has been working on an Object File System for years and it is rumored that it might finaly ship in Yukon.
A database baked file system is a great idea for an O/S. But the relational model is long overdue for the garbage pail. Modern programming languages since C have used pointers or object references. If JOIN and messing arround with tables is so good why don't we all use COBOL?
One of the things that appeared in VMS a while back that was pretty cool (and pretty easy to do on a log based file system) was transactions at the file level. You could take any set of file I/O operations and wrap a transaction arround them. This meant that you could have atomic updates to any file base resource without having to suffer the pain of SQL.
It would be pretty easy to implement this on a Linux log based file system (or windows for that matter). All you do is extend the log structure so you can group operations together and implement some sort of commit flag.
You could then build an object oriented filestore database using XML flat files. OK so maybe the system is not going to be up to storing millions of records without more infrsastructure. However most programming tasks use configuration files that are unlikely to be more than a few tens of Kb and are routinely managed as in memory structures anyway.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
It seems to me that the majority of people who reply with "I use HFS just fine, file-> save as -> my documents works just fine" are also the type of people who don't actually create more than a few documents anyway.
I write a lot of documents and my filing system becomes ever more difficult to manage, without the skills that a librarian or filing secretary has I find that my documents become harder to locate over time. To me this is a potential solution to that problem, I do however appreciate that "Joe Bloggs" will not understand what it is about, but as far as I am concerned "Joe Bloggs" should not be using computers in the first place. Pandering to his ilk has set computing back 10 years.
The potential pitfall of this system could be where many documents have been written about the same subject i.e. testresult001.txt to testresult999.txt. The user would know with the traditional system that he wants testresult823.txt but with the new system would be presented with 1000 choices. I am possibly being myopic here!
Perhaps it is time for a new paradigm and I for one will be looking at this method with great interest.
I have used "The Brain" while I was in Windows, but it was nearly useless as it didn't support the two most important things:
.docs or .xls.
a) Web browsing
it should now the sites you've visited, know your bookmarks and allow you to open everything you have found with a simple click.
b) E-Mail.
When it finds an E-Mail a simple double-click should be enough to open it in your mail, show you the thread it belongs to, etc.
I guess, that I'm not the only one, who has more important things in mails than in
Bye egghat.
-- "As a human being I claim the right to be widely inconsistent", John Peel
But this is the first time I've seen it implemented in userland.
Re: submitter's cockiness about innovation, I think it's simple a pumped up way of saying "if I hadn't have had the source, I couldn't have done this hack". No shit.
Maybe it's just me, but I think it would have been truly more clever if it had been implemented using a stacked filesystem, or even a hacked open(2).
So where do your documents go when you save them with newdocms? As you might have noticed (if you looked at the window titles after saving something), they are stored as ~/Docs/{numeric id}.{ext}.7 All the metadata is stored in a file called ~/newdocms.db. (It is not wise to delete it!) In that file each document's attributes are associated with its unique numeric id (the one which is used as a file name).
Right.
This is astoundingly bad software engineering.
Manuel, when your software fails, and it will, and somehow that db file gets trashed you've rendered that users' files as a huge heap of unsorted data. Effectively it would be 100 times worse than never implementing your system than 10 times better. No matter how bulletproof you think your code is, it probably isn't 100% perfect so having all your eggs in one basket is unwise to say the least.
Even if your code is 100% perfect this is a mistake. What happens when a sector goes bad and this file is trashed? What happens when the first really dangerous linux worm makes it a point to delete *.db from the filesystem?
Give the files names that are coded with human readable attribs! Double up that db file! Jesus, man... build SOME kind of redundancy in your system before you throw away the old way of storing the data.
There's a reason why there is such a scramble to implement a general attribute system at the FS level on many FS projects right now(*). The time has come for OSS to start being smart about this, but cramming all your metadata into a single file and throwing the backup out the window is just a very, very poor idea.
(*) BeOS was, yet again, way ahead of it's time with BeFS.
Consider this: you save your spreadsheet today as "Yearly Report 2002", and two days later you want to call it back your mind just doesn't say "Yearly Report 2002", but more like "Financial Data last year".
um, he took care of this.
try reading the article next time.
(am I feeding a troll if they're marked +2 Interesting?)
I believe metadata is a useful additional means to find files, however I would still want heirarchy as the primary storage. For most people the only metadata they ever consider is the name of a file, and this is often poorly named. I applaud the effort of the person who is doing this project though.
-- Solaris Central - http://w
What is the deal with people wanting EVERYTHING in a SQL/LDAP style databse! Every intern I have to manage out of college seems to have been brainwashed to think that whatever the app, it's data should be in a relational database.
I like datbases, but for somethings they should not be used!
When it comes to the OS, I want to be able to text process data EASILY...with BASH! This road leads to things like binary configuration files and that leads to things like the Microsoft registry which I detest.
Databasizing everything (including the filesystem) IS NOT THE ANSWER
On slashdot saying the emporer has no clothes is considered "trolling".
"it is a layer between the hierarchical file system (HFS) and the user, which provides a radically new way to store and retrieve documents"
The only things that are radically new is that a) it is open source b) it is aimed at individuals.
Commercial EDMS (Electronic Document Management Systems) vendors have been doing this for years - companies like Documentum and Filenet. Like newdocms, they combine the filesystem and a database of attributes for file storage and retrieveal. Documentum itself has gotten very sophisticated; it can read the text of the document and, using an XML taxononmy of your choice, auto-file the document in the appropriate place. Likewise searches can be performed on both user-supplied attributes, computer-generated taxonomic values, or the text of the document itself. Companies like pharmaceutical manufacturers, with millions of complex documents, simply couldn't function without it. And it works with multimedia files as well, actually reading the close-captioning on movie clips to perform it's auto-tagging and auto-filing operations.
That said, it is a great accomplishment, and a welcome addition to this KDE user. While I have worked with EDMS systems off and on for the past 5 years, I could never actually afford one myself, and none of the current EDMS vendors that I'm aware of support Linux.
As work begins on version 1.1, I suggest taking a look at the features available with the commercial EDMS products for ideas - especially for things they haven't thought of!
Good work!
Here's something that is not stressed enough in school: the HFS is a database, with the fully qualified path name as unique ID and basic operations of update, delete, record lock, and retrieve supported across most operating systems.
Other query operations are supported such as wildcard characters and, in large OSes other than Unix, a variety of other attribute queries (a la "/usr/bin/find" but accessible from "ls").
Now the file table itself is a database, which can be readily implemented using a relational database. Microsoft NT an other OSes have had such support for quite a while now.
I'm glad to see the full relational database FS model starting to hit the mainstream. By this time researchers are looking into XML based File Systems (store metadata in XML-like syntax, support any XML query on the files).
Which brings us back to an often overlooked fact. Linux has, in general, not been at the leading edge of OS research (with the possible exception of the beowulf architecture). This is alright as for many years the goal of Linux was to reimplement Unix on the intel x386 architecture. However we must keep in mind that the really advanced OS features out there have yet to make it into Linux, things such as new environment metaphors, persistent data support, and intelligent user interactions.
Standard Slashdot Clue-Slap #4: The Fallacy of Mass Hypocracy
If you walked into PNC Park during a game, and saw a group of 10 people wearing Braves jerseys, would you call the remaining 38,000+ Pirate fans* in the crowd hypocrites? What about a vegetarian eating a salad at a steakhouse?
What you're observing is not hypocracy on the posters' part. They're willing to join the debate, and they deserve credit for that. (You imply that much with your preemptive taunt to anyone who would mod you down.) It's just human nature getting the best of the moderation system. It's too easy to silently and anonymously squelch a valid dissenting opinion. And while meta-moderation can cull out the egos and zealots, it operates too slowly to keep up with short tempers.
*: Jokes about the Pirates selling out a home game > /dev/null :-)
This sig intentionally left blank.
I think what you are missing is that human beings find categorization intuituve - not the hierarchy especially. A hierarchy is simply just one form of categorization. This becomes immediately obvious once you try to categorize things by multiple attributes. For example, I'd like to categorize my mp3s by author, album, genre, and year. That is a perfectly natural and intuitive desire, but it can't be achieved with a simple hierarchical file system.
yes, this system does take care of this. you're limited in your thinking, you're still looking at this from a HFS point of view.
you save your spreadsheet today as "Yearly Report 2002"
the thing is, you don't save it as "Yearly Report 2002". that's a file name, which is not what this system does. in newdocms, you give as much information about the document as you want, in any number of categories, and you don't have to remember arcane names, or the difference between "Yearly Report 2002" and "reports/yearly/2002". in your situation, after 2 days have passed, you come back and make a simple query for, say, Reports, and there your document is, right at the top of a list (since it's sorted by last access time).
I'm 21, started with DOS at age 8, so I can handle hierarchical file systems. that doesn't mean I have to like them. in my case, especially; I'm extremely messy (room, car, apartment, desk), and my hard drives are the same way. I have around 180 gigs spread between 7 drives, and I have no idea where anything is. I could find everything and categorize it in a hfs, but it sure would be easier to not worry about details like "where a file is located" or "what a file is named" and just worry about the types and contents of the files.
Oh, and what if he wants to find the "Yearly Report 1976"? Guess that one won't be on top of the list. I guess this system is good for recently used files, but not for files that have been archived for years on your machine. (Yes, I do have files back from 1981, and yes, I know exactly where they are)
And that you are messy is your problem, not mine. Learn to organize, makes your life easier.