'Storage' to Replace Traditional Filesystems?

JigSaw writes "OSNews is reporting on Storage, an innovative project which aims to replace the traditional hierarchical filesystems with a new document store which is database-based (PostgreSQL). The current implementation, built under Gnome 2.x for now, offers natural language access, network transparency, and a number of other features. The project is currently in alpha (screenshots already available), and it is part of the next major generation of Gnome. It is currently developed by Seth Nickell, the person responsible for the enhanced Gnome usability on 2.x and its HIG, among other things."

Comments

so is everyone copying BeOS

[Anonymous Coward]

It's really a sad that there was a perfectly good implementation of database file system, but the company wasn't able to topple a monopoly and got squashed. MS really should have just bought BeOS and ported everything over to it. They could have just called it LongHorn and released it this year instead of waiting until 2006.

Re:so is everyone copying BeOS

[Twylite]

Summary of developments:

• BeOS has a good idea
• Microsoft announces a breakthrough in file system technology (around 1996), nothing happens
• newdocms announced on Slashdot in January 2003. Integrates with KDE, so no-one cares
• Microsoft announces WinFS plans for Longhorn. Slashdot decides that Microsoft sucks.
• Initial release of Haystack from MIT. Screenshot has XP interface so no-one gives a toss
• WinFS is reviewed, Slashdot has a flame war about file system layout, and concludes that MS sucks and a database file system is a stupid idea anyway and no-one wants one
• YEDFS (Yet Another Database File System) announced calling itself "Storage". Integrates with GNOME. FLOSS community bows and worships the superiority, leadership and sheer innovativeness of the application.

Re:so is everyone copying BeOS

[jilles]

Be-os deserves some credit for merging meta data with a file system. However, a real database goes a few steps further in terms of the ability to query, to do replication, remote data access etc.

Essentially, the Be-OS filesystem, while much richer than other filesystems, is still a filesystem. This Storage thing is a full blown SQL database in the first place.

Essentially a normal filesystem is a hierarchical database where as modern databases are relational or object databases. Relational databases have proven themselves for storing complex data over the past few years.

Some scenarios to give you a clue as to why the distinction matters: you can set up a database trigger to track changes in wordprocessor documents (i.e. automatically update some table with version info whenever you click save); you can involve external databases when doing a query on your own database (e.g. the imdb example in the Storage proposal, a tv guide); emulate a hierarchical file system by associating directory attributes with an object; emulate multiple orthogonal hierarchical filesystems; integrate security policies and encryption into the database (could also be used for DRM, I know this is a sensitive topic); make the objects themselves database records (e.g. contact information); use report generators and queries to dymamically generate complex documents (e.g. software documentation, financial overviews, etc.). Use special purpose software to browse specific types of information (e.g. a picture album, movie library or an old fashioned filebrowser).

Finally something new to play with!

[Trigun]

Hopefully they plan on extending this to the networked environment, allowing multiple domain/realm file permissions, authentication, and encryption.

Anything to replace NIS and its bastard stepchildren.

Obvious advantages

[tsetem]

There's lots of advantages to this kind of system, especially if interfaces are written for other OS's (Windows, Solaris, OSX)

• Networked file system. No more NFS/SMB hacks. Everyone accesses the data in a common way, and can access the same data
  
• Integrated mime-types. No more relying on file extensions and other hacks. The mime-type (and subsequent viewer) is right there in the query
  
• Integrated version control. Have and keep a history of all of your files as they were managed and maintained through their lives, as well as a history of who modified them. If this aspect could be enhanced with branching & merging, then would make other CM Systems (CVS, ClearCase) obsolete?
  

Of course it's only wishful thinking. I'd be nervous to see exactly how this integrates into other "Legacy" applications. I can also see be performance penalties since you are now querying a database, rather than looking at a simple file structure...

Re:Obvious advantages

[dabadab]

"Integrated mime-types. No more relying on file extensions and other hacks. The mime-type (and subsequent viewer) is right there in the query"

And how does that meta data gets to the db? Oh, right, it will rely on file extensions and other hacks :)

Re:Obvious advantages

[azaroth42]

Obvious disadvantages:

SQL is slow compared to things like BerkeleyDB

We already have journaled file systems that can save metadata (though not user defined, I think)

Your database becomes corrupt, you lose everything.

Sorry, give me something that gives me back my data -fast-. If I want to do selects for files, I'll use locate and xargs.

--Azaroth

Re:Obvious advantages

[laird]

"And how does that meta data gets to the db? Oh, right, it will rely on file extensions and other hacks :)"

Like it has in MacOS for 20 years -- when applications write files, they tell the OS the filetype. The only time MacOS looks at extensions is if it's dealing with files transferred from operating systems that don't have relevant metadata. Unfortunately, that would be nearly every other OS. :-) But if Linux started transferring filetype metadata that would be a nice step in the right direction.

How does the metadata get into the database?

[farnz]

My major concern with all these database type filesystems is that the gains are always shown as things like, "Find all films directed by Steven Spielberg", and yet this is not information that the computer can necessarily gather for itself.

Outside of a work environment, I've rarely encounter anyone who keeps consistent, useful filenames, let alone metadata indexes; it seems to me that people will skimp on the metadata, and thus limit the usefulness to metadata that the computer can collect automatically ("All movies that last under 90 minutes"). It's like CD collections, or books; libraries have nicely catalogued and ordered collections. Private individuals don't; they have roughly ordered collections on the shelf, and don't bother keeping them in any better order. I suspect the same will happen with these metadata systems; people won't do the work needed to make them truely useful.

Re:How does the metadata get into the database?

[henbane]

"It's like CD collections, or books; libraries have nicely catalogued and ordered collections. Private individuals don't; they have roughly ordered collections on the shelf, and don't bother keeping them in any better order"

Call yourself a geek? How can you possibly but something on a shelf without first checking to see that it's in a proper place observing the subtle cross reference system that backs up the obvious system. Man, I hate it when people move my stuff.

Re:How does the metadata get into the database?

[Tom]

That's why we have community products. For music, CDDB works pretty good and is a working real-life example.

Other metadata is automatically inserted. When you install OpenOffice, it asks for your name and inserts that as the author into any new documents you create, for example.

Sure, the metadata on my personal machine will never be comparable to what a library could do. But it doesn't have to be - it has to be useful for me, not - like the library - for thousands of people with very different interests and approaches.

Re:How does the metadata get into the database?

[Anonymous Coward]

There are two kinds of metadata: intrinsic and extrinsic. Intrinsic metadata, as the name would imply, is information that's contained entirely within the file.

Some intrinsic metadata can be extracted with automation. For example, it's pretty trivial to examine a TIFF and tell you that it's X by Y pixels in a given color space. It's harder, but still possible, to tell you that the TIFF is predominantly red and green. It's impossible for the computer to tell you that it's a picture of a barn.

The same is true of extrinsic metadata: some of it can be extracted automatically, but not much. An example of extrinsic metadata would be a relationship. The computer can tell you that main.c and somefunction.c are both C language source code files, but it may or may not be able to tell you that they're both part of the same program. If the two files are explicitly related to each other through a makefile or some such, then the computer can know that they're related. But consider the collection of JPEGs I just copied to my home directory from my digital camera. A dozen of those pictures were taken in Fiji. The computer cannot know this unless I tell it, nor can it know which pictures were taken in Fiji and which were taken in my back yard last Tuesday. Thanks to my camera, the computer can know what apeture and focal length were used for each picture. In theory, if my camera had a GPS receiver in it, the computer might even be able to tell me there, on earth, the camera was when it snapped a given photo. But these are just automatic methods of telling the computer about the pictures. They're conceptually no different from sitting down and typing the information in. The point remains that the computer can figure out some things on its own, but it cannot know most things unless it is told.

You don't have to strain your imagination to think about this stuff. Consider your MP3 collection. Your computer can tell you that a given MP3 is 6:15 long, and that it's 192 kbps, and that it's stereo. It can't know that it's "Treefingers" by Radiohead unless somebody tells it first.

So you're basically right: automatically extracted metadata is marginally useful, but the really useful stuff has to be manually entered. And generally speaking, even in business environments, that sort of information simply never makes it into the database. It exists exclusively in people's heads.

That's the biggest challenge of digital asset management--which is, incidentally, essentially what we're talking about here. The biggest challenge is how to take information that people have in their heads and store it in some structured, persistent form. That form might be a three-ring binder or a card catalog or an Oracle database; the challenge is the same even if the technology is different.

Bottom line: this technology is really neat, and has limited applications in which it's very useful. But it is not generally useful, nor does it have widespread applications.

Re:How does the metadata get into the database?

[selderrr]

Very true and insightful !

Another argument to prove you right is the "rate this song" options in iTunes. With that feature, one can assign 1 to 5 stars to a song so that later, you can quickly select your favourites. Such system is flawed in 2 ways :

- I have yet to encounter anyone who uses it exhaustively. Most folks rate a few dozen songs. I have a library with 9000 mp3s and sure as hell I'm not going to spend a whole week rating them.
- I have yet to encounter someone who uses it consistently. Today I might consider Chris Isaak a 4star song since I'm in a depressed mood and it's raining outside. Tomorrow the weather might be beautifull and I mod him down to 2 stars cause he's a bloody negative wanker.

This is offcourse iTunes specific, but it shows that the assignment of metadata is far far far more complex than the methods to search/organize the stuff, which is what the "Storage" software above is about.

As an extra complication : consider that my metadata might not match someone elses. For instance if I were to label a mail "message from my brother", the same content would be "message from my son" for my father !

The fact that metadata based filesystems are not on our desktop is perhaps more to the fact that it's not a valid solution for data on desktop computers. Maybe, just maybe this is not due to MS squashing Be, as someone else was karmawhoring above.

Re:Hmm

[UnuMondo]

No, because doing away with the root filesystem, user stuff in /home, config files in /etc, and so forth would break a number of Unix standards Linux's big advantage of being able to run many Unix apps (if you compile from source) would disappear. Storage will apparently be an interface to the existing real filesystem. Joe User won't know the difference.

Limitations in the home edition

[yerricde]

What then happens to SQL as a MS product? If its built in to every OS, why then would anyone buy it.

Remember how Windows XP Home and Pro editions can serve files only to less than a dozen simultaneous clients? This is to boost sales of the IIS bundled with Windows 2000 Server and now Windows Server 2003. Microsoft SQL Server Home Edition will probably be limited.

AS400 did this 20 years ago:

[mikepb78]

The filesystem on AS400 is actually a db2 database and it work quite well

Re:Windows?

[Zocalo]

Not quite, NTFS is a traditional file table with some bells and whistles, but it's not a "database" in the sense meant here(1). The next version of Windows, "Longhorn", is supposed to introduce a new file system called WinFS that will use a version of SQLServer as its backend. Whether they will actually deliver or not is another matter, since we were promised this in 1995 with Cairo and Taligent (remember them?), and now that Longhorn appears to have been pushed back...

There are also issues with gaining acceptance for the change in the way things work. This kind of thing has not really been done on a large scale in the wild before, on any OS, so whether people will be willing to accept the security and reliablity issues that may ensue is another matter. For example, what are the implications of a compromise in the database engine? MS is planning on using SQL, so if things go awry and it becomes possible to maliciously inject raw SQL to the filesystem interface... Oops. On the otherhand, the benefits for data retrival are *huge*. Imagine being able to find any audio files on your entire system by Justin Timberlake or Britney Spears and delete them all in one go by searching on the tag fields! ;)

(1) Technically, all filesystems are databases, it's just that current ones are a collection of flatfile database tables that can point to each other, generally in a heirarchial manner. When people say "database" in the same sentence as "filesystem" they usually mean "relational database". As an aside however, high end databases usually forgo the need for a file system and provide the ability to write their tables directly to disk on a dedicated partition.

"Damn, I left that on my roommate's desk"

[kfg]

"Well, where do you go?"

"Stanford."

"No problemo, I'm heading that way later and I can grab it for you. What's your room?"

"Dorm 5, Room 109. It's the desk on the left."

( We didn't bother to state earth.us because we were already inside those directories)

Yes, yes we do think heirarchically. Most of the history of human thought has been fitting everything we can lay our filthy little brain cells on into heirarcheis, whether they wish to fit into them or not. It's intuitive.

As for natural language didn't we learn about that with COBOL? Natural language only speeds the learning process slightly ( the majority of the learning still lying in the realm of understanding the basic concepts involved), but then becomes a pain in the ass forever afterward.

Looking at the screenshots it's also ugly as all sin. The physicist in me can't help but feel that a model that ugly can't possibly be correct.

I think this makes just about as much sense as using a document preperation language (XML) as the basis of a database.

Which is to say, none.

KFG

Re:Windows' filesystem

[Zocalo]

Good guesses. Replace "SCO" with "Apple" and you probably have the right triumvirate. All three were working on this in 1995 or so - Microsoft was going it alone with "Cairo" (should have been Win2K) and IBM/Apple were working togther on "Taligent"/"Pink". Neither project saw the light of day, although whether this was because of the system requirements or a marketing decision based on the paradigm shift is a matter of opinion.

The idea was probably stolen from Xerox Parc in the first place, of course.

Random thought for the day...

[fluxrad]

Am I the only one that isn't totally into the idea of "googling" data on my hard drive?

Granted, it's mostly pr0n on there, so it's almost the same thing, but still...

Not SQL Server Directly

[Watts]

Having SQL Server as the underlying filesystem technology doesn't mean that you're going to be running SQL Server directly. I mean, if you currently use NTFS, there isn't a NTFS daemon that the kernel connects to when it does filesystem transactions. Just like every other filesystem, the support will be built into the kernel. Instead of writing data as NTFS does, the structure will look a lot more like how SQL Server stores data -- with built in indexes, etc.

Many database servers already have some fairly optimized code when it comes to file access. This just implements it at the kernel level, rather than having it sit on top of a traditional fs.

Oracel IFS

[rhinoX]

It was called IFS and Oracle did it like, almost four years ago.

Versioning and various other metadata existed. It could be exported via SMB, NFS, FTP, and as a regular "local" windows filesystem.

And, why is this such a great big deal? I don't see the same stink raised as the possibility of Longhorn having a DB for a filesystem.

Re:i think

[laird]

I disagree, strongly. Files are an artifact of a bunch of bad implementation decisions when stripping Multics down to produce UNIX. What programmers want to be able to do is manipulate data structures and store them persistently. What files force you to do is waste tons of time writing code to take your data structures and write them out as sequences of bytes and read them back in.

One OS that solved this nicely was NewtonOS. If you wanted to manipulate persistently stored data you opened a "soup" that contained objects. So if you wanted to, say, set up an appointment with someone for lunch, you could find the person in the address book "soup" and then create an entry in the databook "soup" recording the appointment, which would immediately appear in all other apps that dealt with appointments (because app's accessed the same data structures, and were notified of changes so that they could update). So your data was not trapped in a particular application's proprietary format, and users weren't forced to learn the artificial concept of a "file" but instead could think about "my appointments" or "my address book".

If you haven't tried it, don't knock it. As a developer, and as a user, it was wonderful -- much more straightforward than "files" and "directories".

Microsoft Attempts for decade,GNOME Does in months

[NZheretic]

1994 Cairo Takes OLE to New Levels

The next version of Windows NT, code-named Cairo and targeted for release sometime in 1995, will be built around the concepts of objects and component software. It will have a native OFS (Object File System) and distributed system support.

1995 Signs to Cairo

Cairo, Microsoft's object-oriented successor to Windows NT, will begin beta testing in early 1996 for release in 1997. Although Microsoft is not revealing the full details of Cairo yet, there are enough clues within current Microsoft OSes to yield a good idea of how it might work.

1996 Unearthing Cairo

At the first NT developers conference in 1992, Bill Gates announced that Cairo would arrive in three years and would incorporate object-oriented technologies, especially an object file system. Since then, we've seen Windows NT 3.1, NT 3.5, NT 3.51, and most recently NT 4.0. None is object oriented, none has an object file system, none is Cairo. It seems that Cairo is Microsoft's sly way of promising the world. "Will we see Plug and Play in NT?" "Oh yes, of course, in Cairo." "Will NT ever produce world peace and cheap antigravity?" "You bet -- in Cairo."

The so call Longhorn WinFS directory is just another rencarnation of the Cairo object orientated file system.

September 1, 2003 Eweek 'Longhorn' Rollout Slips

Microsoft Corp. has once again shifted the schedule for the release of "Longhorn," the company's next major version of Windows, leaving some users up in the air about an upgrade path.

Microsoft executives from Chairman and Chief Software Architect Bill Gates on down have long described Longhorn as the Redmond, Wash., company's most revolutionary operating system to date. The product was originally expected to ship next year. Then in May of this year, officials pushed back the release date to 2005. But now executives are declining to say when they expect the software to ship.

"We do not yet know the time frame for Longhorn, but it will involve a lot of innovative and exciting work," said Gates at a company financial analyst meeting this summer. Since then, other Microsoft officials have neither retracted nor clarified Gates' statement.

Microsoft have been attempting this type of functionality since 1991, over a decade. Meanwhile, one open source GNOME developer, with help from the other core GNOME developers, provides most of the features within months.

Re:ext3 + sql

[rtaylor]

It won't improve performance if you know exactly what you are looking for. The goal is to improve performance when you only have a vague idea of what you want.

This isn't a place to store config files or cronned shell scripts which have definitive locations and content.

This is a replacement for that 5TB corporate filestore with a 50 directory hierarchy that nobody can figure out, and a content based find takes days to complete.

GREAT! If it is done well...

[evilviper]

People don't seem to see how great this is. Maybe it's because most people don't have all that much data.

On my home systems, I have over 250GB online. That doesn't even count my music or videos/movies, which I keep on seperate, removable, optical storage.

I can tell you from experience, that managing that much data is a huge hassle. Let's say you've got your files organized well. You probably have hundreds of folders for each subject, and you have to broswe to each one with each new file you save. I have a folder (several actually, for various subjects) where I save thing that I've haven't taken a look at yet. Let's say it's a program that I haven't installed. Well once I do install it, I need to clean up all the temporary files, then browse around to another folder (takes a minute or two when you have hundreds of folders), where I save installed programs, and browse to the appropriate sub-folder, and save it. But then I end up doing the same thing with a video clip... Watching it, deciding to delete or save it, then browsing to a sub-sub-sub-sub folder to move it.

Of course, that's enough of a hassle, but things get complicated when I want to move things to another systems, which obviously isn't going to have the same filesystem. Merging each individual folder, into each different folder is seriously time-consuming, and teedious. Without fail, there always ends up being a couple folders in the wrong place, because they were a sub-folder of something else, that I did happen to see when I coppied the contents of the folder.

Then matters are even further complicated, because I may choose to delete older content months later or so, and locating everything is a huge mess.

Personally, I would like to save everything in one place, not having to change folder to folder for each file. When saving something, I could just enter a handful of keywords (eg. "picture penguin snow") which would be much less work than moving to directories or even typing in a long filename. From there, a simple database system would be be able to know what type of file it is, how large it is, and how old it is. That would make it incredibly easy to manage. Whenever I want a file, I type-in "images older than 1 years" or "programs marked as archived" and I get EVERYTHING I'm looking for in a fraction of the time. Not only that, but it makes pruning out old data as easy as it could possibly be. Just search for "linux" and delete older version, no worries about what folder it's in... If it's in a temporary folder and you haven't used it yet, or if it's archived and been in-use on your system forever. Obviously you'll be able to see that information, but unlike in our current systems, it won't stand in your way when you want to find things.

It's absoultely no work at all to transfer files, since the info should stay with them, and it will automatically integrate perfectly with your local file management/organization scheme. What's more, data like marking something as "archived" is great in that your system could automatically move it over the network where you archive your files. Since your filesystem would be a smart database, when you search for the file, it could still turn up in the search results, and be automatically moved back where you need it, when you need it.

Personally, I think this would not only save time and effort, but money as well, because so many people wouldn't be dealing with their file problems by just throwing more space in their systems, instead of spending time on figuring out where every file is, what they can get rid of, dupilcate files, and junk like that.

With this, I should be able to say "tar -xjf 'newest version of mplayer'" However, this will need to be in the actual filesystem to be useful, not just supported for GNOME applications.

Nope

[varjag]

> SQL is slow compared to things like BerkeleyDB

BerkeleyDB is a hierarchial database. SQL is godzillion times faster on complex searches.

> Your database becomes corrupt, you lose everything.

Your filesystem becomes corrupt, you lose everything.

And yeah, I know about journaling, so don't bother :) But modern RDBMSes have integrity control facilities as well.

Re:Nope

[Saint Stephen]

Just wait till you see the way "Pivots" work in the new Longhorn shell. The canonical example is sorting thousands of mp3s by artist, but it'll be A-FUCKIN-MAZIN.

Face it: databases rock. You never know how many interesting questions you didn't ask because you couldn't think in sets until you do it, and then it's FAST as all get out.

Re:Replacement for ls

[Mwongozi]

SELECT * FROM Users WHERE Clue > 0
0 rows returned

Ah, humour.

Re:Windows' filesystem

[Pfhreakaz0id]

My guess is it will be something like the MSDE engine. So it will be limited. For those who don't know, MSDE is just an embedded, single-user version of the SQL engine. I worked on an app once that used it for laptop users who were offline from the network and would have a copy of the database to search and enter orders in, which would auto-replicate with the master SQL server when it got back on the LAN. It was pretty neat.

Not exactly

[gilesjuk]

http://theregister.com/content/4/30670.html

"The oft-misunderstood Windows Future Storage (WinFS), which will include technology from the "Yukon" release of SQL Server, is not a file system," reports Thurrot. "Instead, WinFS is a service that runs on top of - and requires - NTFS."

Ah yes, the infamous relational filesystem...

[Millennium]

Although this is an interesting idea, an all-relationsl filesystem would prove to be a usability nightmare.

The relational zealots are quick to point out that a relational system can model any sort of data. Indeed, it can do this. This does not, however, mean that it's always good at doing this. Sometimes it's the right tool for the job, and sometimes it's not. In this case, it is very much not a good tool for sole access to files on the system (though it can make an excellent tool for complementary methods of access).

The reason that hierarchical filesystems have survived for so long is due to one thing: navigability. It's relatively easy for any user to browse what's on the system and get a good idea of how it is organized.

You can't navigate a relational system, which will prove to be the downfall of any all-relational system which comes into being. You can, of course, do a SELECT * FROM volume if you really want to, but that does exactly that: it gives you all the data, with no particular organization. Examining the entire "sea of data" suddenly becomes cumbersome in the extreme. So while User A might be able to set up an all-relational filesystem completely according to his own tastes, User B will be totally lost on that same system. This is, to say the least, a nightmare for anyone working in a shared environment.

This is not to say that the relational model isn't necessarily a useful thing for filesystems. On the contrary, it can be very useful for certain kinds of searches. As time goes on, I believe we'll see more relational-style searching technology incorporated into file managers and search tools. However, there also needs to be a means of hierarchical navigation. Humans tend to think of things in terms of locus, and a means of providing that kind of reference point have to be maintained.

Luckily, this can actually still be emulated using relational-style tables, even though it's somewhat less efficient than classical storage techniques. Some filesystems already do something similar to this, and the results are promising. Look at Be's filesystem for an example of that.

The best way to go, moving forward, is something not unlike what BeOS did, with both hierarchical and relational methods of examining data. This allowed for the best of both worlds. The default method of getting at data is still the hierarchical paradigm, but relational searches can be applied to create what some have called "smart folders" (perhaps "boxes" might be a better term?) Systems like this "Storage" should be focusing on complementing traditional systems in this way, rather than replacing them.

Re:Replacement for ls

[sharkey]

SELECT * FROM MY_FILES
WHERE TYPE = 'video/x-mpeg'
AND TITLE IS LIKE ('*tit*, *blonde*)
ORDER BY PERV_RANK

Re:Windows' filesystem

[simonecaldana]

> The filesystem will be based on SQL Server 2003, but it won't be a fully functional version of SQL Server.

you mean it will be a standard version of SQL server? :)

why the relational model is not right

[hansreiser]

www.namesys.com/whitepaper.html describes why the relational model is not the right one for large heterogeneous stores (filesystems), and describes the approach ReiserFS (a Linux filesystem used mostly in Europe) is taking instead.

Hans

Only as good as your data

[kstumpf]

One thing I haven't seen mentioned yet is that a filesystem of this type is only useful if there is quality metadata accompanying every file you expect to find. Searching for "all jazz music" would return nothing unless the filesystem was told about each file that qualifies as "jazz music". What if I wanted to be more specific and say "jazz horn music"? Even more specific, "jazz trumpet solo"? The filesystem would have to know all of this data to be effective.

Where does this metadata come from? I assume I have to enter it myself. This means the more files I have, the more detailed and specific my data entry becomes. And that much more tedious.

Even worse is the uncertainty that would arise. Is my search for "horn solos" not returning results because there are no such files, or because the filesystem does not have meta data describing the files I want as such?

At this point, hierarchial organization once again becomes much more appealing again.

Comments from Seth (aka Storage's designer)

[nullity]

I suppose it is probably too late to inject comments and have them moderated to the point of visibility as the madness has largely subsided... but here's to futile acts ;-) I was not really intending Storage to make a big splash right now, I wanted to keep it low-key, but I guess the damage is done so I might as well comment. I'm sorry that I didn't have time to put up a more technically-oriented exposition of Storage. *shrug*

• Slashdot has focused almost exclusively on the "database backing". Guys, this is an implementation detail. Its an important one, but I didn't start off this design thinking "lets write a database backed filesystem store". A set of design goals was established (largely mirrored in the features page). Storage is a lot more than just a database backed XML store. Please read the features page. The "searchable" stuff is nice, but equally important is providing persistent objects, uniform access (the same URI for a local storage node works globally assuming your computer has a publicly accessible IP address), an improved model for revision and "saving", the ability to localize filesystem resources, and due to a standard object format greater transparency of filesystem resources to the OS which will be useful in weakening the barrier between "apps" and "desktop" found in PCs (and not so much in, say, cell phones and pdas). This is also a key piece in an overall design of the desktop's interaction structure which I haven't had time to write up for the web.
• I'm not trying to make any claims to being the first or being highly innovative, but I am happy to make claims about improving the user experience. That said, contrary to what people are saying, to my knowledge other than the superficial layer of database backing, Storage's features do not have a "one to one" correspondence with any existing system, BFS and the only vaguely specified Windows Future Filesystem included. Most importantly these components do not seem to be a part of the same overall interaction design model that Storage is intended to support. Storage is just a stepping stone, albeit a pretty disruptive one.
• I've been quiet about this project, even inside GNOME. Storage as written today was primarily written by a team of Stanford students as their CS senior project. I've since been working with a few good GNOME developers including the person working on Medusa (Curtis) and the Epiphany maintainer (Marco). They were independently developing a metadata system for GNOME, which it looks like we may implement on top of Storage as a first major test of its capabilities. But nothing is certain right now. But the short story is that although storage is being developed by GNOME developers and I serve as usability project lead, its not an official GNOME module at this point. GNOME developers would need to corporately buy into both the Storage vision and the overall desktop design. This may never happen, and if it does, its going to be very slow in the coming.

Some technical notes... that site is sparse on technical information so I'll fill in some for the curious.

• The data store is backed by Postgresql. Postgresql rocks, though some of the features like instant notification of object changes and live queries do not fill well with existing SQL. We have ways to do all of this using Postgresql extensions, but sometimes its a little tricky and/or hackish.
• A lot of the proposed interface will rise and fall based on the quality of the NL processing. Storage is currently using some pretty cutting edge linguistics theories and tools... notably working within the basic LinGo framework. This includes using theories/systems like HPSG (Head-Phrase Structure Grammar), MRS (minimal recursion semantics), and being able to use a set of existing wide-coverage grammars such as the ERG (English Resource Gramm