'Storage' to Replace Traditional Filesystems?

JigSaw writes "OSNews is reporting on Storage, an innovative project which aims to replace the traditional hierarchical filesystems with a new document store which is database-based (PostgreSQL). The current implementation, built under Gnome 2.x for now, offers natural language access, network transparency, and a number of other features. The project is currently in alpha (screenshots already available), and it is part of the next major generation of Gnome. It is currently developed by Seth Nickell, the person responsible for the enhanced Gnome usability on 2.x and its HIG, among other things."

Comments

Windows' filesystem

[mic256]

I think Longhorn will be the first Windows with a database filesystem. It will probably be based on SQL Server

Re:Windows' filesystem

[cyclist1200]

The filesystem will be based on SQL Server 2003, but it won't be a fully functional version of SQL Server.

Re:Windows' filesystem

[Pfhreakaz0id]

My guess is it will be something like the MSDE engine. So it will be limited. For those who don't know, MSDE is just an embedded, single-user version of the SQL engine. I worked on an app once that used it for laptop users who were offline from the network and would have a copy of the database to search and enter orders in, which would auto-replicate with the master SQL server when it got back on the LAN. It was pretty neat.

Re:Hmm

[UnuMondo]

No, because doing away with the root filesystem, user stuff in /home, config files in /etc, and so forth would break a number of Unix standards Linux's big advantage of being able to run many Unix apps (if you compile from source) would disappear. Storage will apparently be an interface to the existing real filesystem. Joe User won't know the difference.

Limitations in the home edition

[yerricde]

What then happens to SQL as a MS product? If its built in to every OS, why then would anyone buy it.

Remember how Windows XP Home and Pro editions can serve files only to less than a dozen simultaneous clients? This is to boost sales of the IIS bundled with Windows 2000 Server and now Windows Server 2003. Microsoft SQL Server Home Edition will probably be limited.

AS400 did this 20 years ago:

[mikepb78]

The filesystem on AS400 is actually a db2 database and it work quite well

Not SQL Server Directly

[Watts]

Having SQL Server as the underlying filesystem technology doesn't mean that you're going to be running SQL Server directly. I mean, if you currently use NTFS, there isn't a NTFS daemon that the kernel connects to when it does filesystem transactions. Just like every other filesystem, the support will be built into the kernel. Instead of writing data as NTFS does, the structure will look a lot more like how SQL Server stores data -- with built in indexes, etc.

Many database servers already have some fairly optimized code when it comes to file access. This just implements it at the kernel level, rather than having it sit on top of a traditional fs.

BeFS

[laird]

Actually, Be had two flavors of "filesystem as database" in widespread deployment. OK, not as widespread as Windows, but certainly thousands of users. The first version of Be's filesystem, by Benoit Schillings, was very database like, but performance was so-so. The second version of BeFS, by Dominic Giampaolo, was less general in implementation, but had the same metadata-driven capabilities. There's an interesting article on this at http://www.theregus.com/content/4/24485.html. Basically, Be did everything that this project is talking about, years ago. That's not to take anything away from the project -- it's cool if more mainstream operating systems catch up to the innovations of niche players, because more people benefit. Dominic is working at Apple, so there's hope that MacOS X's filesystem will start incorporating the rich-metadata, dynamic view model of the world. And while MS has (I think) pushed the "filesystem as database" out of the next version of Windows NT/XP/whatever, it's still planned for the next version after that, so perhaps in a deade or so we'll all be able to do what Be did back in '91. And of course, Palm owns the Be code, so perhaps PalmOS will lead the way?

Oracel IFS

[rhinoX]

It was called IFS and Oracle did it like, almost four years ago.

Versioning and various other metadata existed. It could be exported via SMB, NFS, FTP, and as a regular "local" windows filesystem.

And, why is this such a great big deal? I don't see the same stink raised as the possibility of Longhorn having a DB for a filesystem.

Re:Why link directly againsat libpq?

[rtaylor]

Their feature list say it will work with Oracle and other SQL99 compliant databases, so I would assume it isn't linked against libpq directly.

Re:ext3 + sql

[rtaylor]

It won't improve performance if you know exactly what you are looking for. The goal is to improve performance when you only have a vague idea of what you want.

This isn't a place to store config files or cronned shell scripts which have definitive locations and content.

This is a replacement for that 5TB corporate filestore with a 50 directory hierarchy that nobody can figure out, and a content based find takes days to complete.

Not exactly

[gilesjuk]

http://theregister.com/content/4/30670.html

"The oft-misunderstood Windows Future Storage (WinFS), which will include technology from the "Yukon" release of SQL Server, is not a file system," reports Thurrot. "Instead, WinFS is a service that runs on top of - and requires - NTFS."

Re:Obvious advantages

[jeti]

You are aware that almost all internet protocols transfer a MIME-type with each file?

why the relational model is not right

[hansreiser]

www.namesys.com/whitepaper.html describes why the relational model is not the right one for large heterogeneous stores (filesystems), and describes the approach ReiserFS (a Linux filesystem used mostly in Europe) is taking instead.

Hans

Comments from Seth (aka Storage's designer)

[nullity]

I suppose it is probably too late to inject comments and have them moderated to the point of visibility as the madness has largely subsided... but here's to futile acts ;-) I was not really intending Storage to make a big splash right now, I wanted to keep it low-key, but I guess the damage is done so I might as well comment. I'm sorry that I didn't have time to put up a more technically-oriented exposition of Storage. *shrug*

• Slashdot has focused almost exclusively on the "database backing". Guys, this is an implementation detail. Its an important one, but I didn't start off this design thinking "lets write a database backed filesystem store". A set of design goals was established (largely mirrored in the features page). Storage is a lot more than just a database backed XML store. Please read the features page. The "searchable" stuff is nice, but equally important is providing persistent objects, uniform access (the same URI for a local storage node works globally assuming your computer has a publicly accessible IP address), an improved model for revision and "saving", the ability to localize filesystem resources, and due to a standard object format greater transparency of filesystem resources to the OS which will be useful in weakening the barrier between "apps" and "desktop" found in PCs (and not so much in, say, cell phones and pdas). This is also a key piece in an overall design of the desktop's interaction structure which I haven't had time to write up for the web.
• I'm not trying to make any claims to being the first or being highly innovative, but I am happy to make claims about improving the user experience. That said, contrary to what people are saying, to my knowledge other than the superficial layer of database backing, Storage's features do not have a "one to one" correspondence with any existing system, BFS and the only vaguely specified Windows Future Filesystem included. Most importantly these components do not seem to be a part of the same overall interaction design model that Storage is intended to support. Storage is just a stepping stone, albeit a pretty disruptive one.
• I've been quiet about this project, even inside GNOME. Storage as written today was primarily written by a team of Stanford students as their CS senior project. I've since been working with a few good GNOME developers including the person working on Medusa (Curtis) and the Epiphany maintainer (Marco). They were independently developing a metadata system for GNOME, which it looks like we may implement on top of Storage as a first major test of its capabilities. But nothing is certain right now. But the short story is that although storage is being developed by GNOME developers and I serve as usability project lead, its not an official GNOME module at this point. GNOME developers would need to corporately buy into both the Storage vision and the overall desktop design. This may never happen, and if it does, its going to be very slow in the coming.

Some technical notes... that site is sparse on technical information so I'll fill in some for the curious.

• The data store is backed by Postgresql. Postgresql rocks, though some of the features like instant notification of object changes and live queries do not fill well with existing SQL. We have ways to do all of this using Postgresql extensions, but sometimes its a little tricky and/or hackish.
• A lot of the proposed interface will rise and fall based on the quality of the NL processing. Storage is currently using some pretty cutting edge linguistics theories and tools... notably working within the basic LinGo framework. This includes using theories/systems like HPSG (Head-Phrase Structure Grammar), MRS (minimal recursion semantics), and being able to use a set of existing wide-coverage grammars such as the ERG (English Resource Gramm