Slashdot Mirror
How To Implement A Database Oriented File System
ALundi writes "A really great read from Andrew Orlowski over at The Register on how Benoit Schillings and Dominic Giampaolo created the 64-bit journaled and attribute based Be File System. Schillings and Giampaolo discuss a variety of design and implementation issues, including data integrity and file system performance. " Interesting in the context of MSFTs plans to
implement a DB filesystem
in future versions of MS Windows.
I used Be on and off for about 6 months. Once you get the hang of it(the filesystem), you see the true power- especially with the address book.
If you want some real insights into the OS as a whole, check out the BEOS Bible. not so good if you want an in depth discussion, but for non-kernel hackers it's a fun read and very informative.
Read the section explaining how their address book works. it's really cool.
Looking for Book Reviews? Check out Literary Escapism.
Does this mean that Apple is finally going to put some kind of reasonably modern filesystem under OS X?
Have they finally seen the true genius behind their own iTunes interface?
Have they finally realized that they will shortly be THE ONLY operating system that still relies on file extensions as the primary way of identifying files?
I truly hope that this snippet is as wonderful as it sounds, as it may finally restore my faith in Apple, as well as cure me of my unhealthy Debian and XFS addiction.Karma: Incomprehensible (Mostly affected by posting at +5, reading at -1, and metamoderating everything unfair.)
for the majority of databases the data should be moved to the filesystem no the database.
Simple joins, and most of them are can be replicated with links if necessary. Almost all the databases I've seen would lose little from moving out of the DB and into the filesystem.
It doesn't scale to complicated joins and huge datastores with complex triggering but for most stuff it simply isn't used.
Too many developers have the mindset of placing tree based data into RDBMS which adds complexity.
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
Object-oriented graph system
- need much less overhead than database system
- can hold multiple ways to access one object, easing semantic link
representation
- can be secure by allowing only the workflow links.
It should be organised as a set of object that hold data within them,
with links between them.
One of the areas that I am interested in is using a DB FS with various "helper" apps. These helper apps would provide a way of managing your data. They could be integrated into another app directly or as a plugin or they could be standalone apps by themselves.
One could provide a helper app that allows you to look at the stored files in ways other than your typical file listing. To do this would require various metadata attributes to be associated with the data. Helper apps would be provided for all of the standard applications that read and write to data to the file.
For example, one app could set attributes to categorize data. One could then search for data on your system, and potentially others, much in the same way as you would search for stuff on Yahoo.
Say you are in a rush to finish your taxes but you need to put together an itemized list of business expenses. You have information on them stored in various places including text files, e-mail, spreadsheets, etc. You could use find and grep to go poking around looking for them or you could use a helper app that does a quick search of attributes and presents you with a list of candidates and ca even call up other apps or services to look at the data. Once you have identified the data another attribute can be set that your tax software uses to record it and pull it into your tax forms.
Here's a past discussion and here's how it's done.
Has anyone actually tried it?
"A mind is a terrible thing to taste."
Is not the new ReiserFS 4 (due for release late summer) going to be a DB FS? Check out http://www.reiserfs.org and read for yourselves.
MS was so proud of their registry that they thought that they could make the whole file system a part of the registry. It broke under stress of actual use, and now the registry is on its way out, never having been much of a success. I expect that the database file system will do the same for a few generations as well.
I checked over the specs for this over at the Oracle site... looks impressive! Now, if only you didn't have to use Oracle to take advantage of this sort of thing; PostgreSQL is standing in first place as my database platform of choice for most tasks.
Is there any kind of equivalent work being done in the OSS community, perhaps based on PGSQL? The platform I'm working on for web-based content management is actually headed toward being usable (and will be licensed under the GPL as soon as that happens), but doesn't use the "real" system FS for storage at all. BLOBs are used for file storage instead, and performance seems to be fairly good
I suppose the best way to go is to structure the system such that it can be used for intranet-style doc/file management, in addition to being able to dish up web-based content (whether for internal or external access/use). We'll see how it shapes up, I guess
Hardware has come a long way since the days of BeOS 4 (I believe that's the last version that used a "true" database backend for the filesystem). With the added power we have these days, it seems the old performance issues might be largely eliminated. Now, I'm not trying to portray myself as an advocate of "throwing hardware at a software problem"; to me, it's really more a case of "we can now do things we couldn't do before." This is somewhat akin to being able to do better 3D modelling on PCs due to new capabilities in 3D hardware acceleration.
If the data is shared, and you have libraries that are shared, then why not ask the data to display itself (object.display(x);) and have it call to a standard library (system library?) which queries a system properties database object as to what application to display it? Don't actually store the display code in the objects, but have the objects query the system as to what the user has specified to display that type of data with.
Dominic: That's what I mean. Some people are very anal about organizing things in rigid hierarchies and others are 'I know what I want to find'.
I think there is a place for hierarchies, but not as the base organizational method of the filesystem. I would like to see a hierarchy of attributes, or keys, or whatever you want to call them. When you save an object (off the internet, or out of your head), a title is only one possible attribute you want to give it. When I save a pr0n jpg, it doesn't need a damn title, I need to mark what it's a damn picture of (amateur AND cumshots AND redhead)! Perhaps start with people, places, things. Or later in the hierarchy, sound -> music -> various bands as well as various artists as well as various sound effects as well as dates and live or studio, all keyed (so to speak) and queryable. But the hierarchy is for browsing. Just for browsing, because browsing is important (when you want to look at cumshots, you want to look at cumshots, but when you query for cumshots, watersports and lesbians, well that's bloody well what you should get), and micro$oft's nice little explorer looks about right. Although instead of a stupid directory tree, we have a tree of object properties and types, and any object can be in any number of places in that tree, depending on it's attributes (categories?).
I know of course that I haven't really said anything new in this post, and I know that performace needs to be taken into account. This is, however, the way things are moving, and all we really need is a really good, really fast, solid state storage medium. When permanent storage is as fast as or faster than RAM is today, the database filesystem will finally become a reality, until then, we'll sure be gearing up.
Cheers, Joshua
When in danger or in doubt, run in circles, scream and shout!
As the ExtremeTech article pointed out, they are not even considering putting the full-blown SQL Server into Windows. SQL Server is too resource-intensive (it really wants to use all of the available CPU, memory and disk space), too much overhead, and most importantly to MS, too profitable (sales of SQL Server / BackOffice make up about 10-15% of MS's revenue.) There's no reason to bundle it if people are willing to pay a ton for it separately!
As the article says, they're thinking (nothing decided yet) about including MSDE, which is exactly the same as SQL Server 2000, except it is tuned for 5 concurrent users (and hard-limited to 10), the database size is limited to 2GB per database (the same as the Jet DB, aka Access), and it doesn't have the nice GUI admin tools bundled.
Also, the OFS (Object File System) discussed previously probably won't get added either. There's a good reason why it was talked about way back in the Cairo (pre-Win95) days but never implemented - it's really, really hard to do, and it's hard to even convince anybody of its value. (Just look at Be.) Active Directory was originally supposed to be an object store, but I don't think anybody uses it for that (if anybody even uses it at all.)
What probably will be included is an improved version of Indexing Service, which is currently included in Windows 2000 and XP. For those of who are fortunate enough to be unfamiliar with Indexing Service (formerly Index Server), it's an NT service (think "daemon") that periodically scans the file system for new / updated files, and then adds whatever metadata it can extract into a database of sorts, which is then used to speed up searches in the built-in Search dialog on the Start menu.
There are a couple of problems with the current implementation:
So, in summary, MS's plans for the DB-in-the-filesystem look a lot more like Reiser4 than like BeFS or SQL Server.
Functionally, database-based file systems are an old hat. If they were the magic bullet Microsoft and BeOS think they are, they would have caught on long ago. To me, it looks more like a bunch of college hackers getting mightily excited about a whizbang feature of little real value. (Database based file systems have worked well in some niche markets--IBM is selling some systems with such file systems.)
Something needs to be done about indexing and search, but putting a database into the kernel is not the right thing.
This whole discussion is entirely wrong in its direction. While the rest of the world is moving towards managing data in a user space, world readable, flexible format that is xml, microsoft is yet again going backwards into proprietary extensions and api's that aren't transferable.
,XQuery, Schemas and xml libraries in general makes me confident that in two years using a xml as a primary data store as well as programming interface will be a breaze. Think about it, what is really missing from xml that a relational database has right now? Basically some indexing scheme and a good api to handle locking and concurrency, other than not really a whole lot. Throw in a little client server and you're done. Now once you've gone that far, what does an object data base have that an xml database doesn't? Not a whole lot, throw in some XPointer stuff, and you've got references nailed.
Sure there might be some speed advantatages in certain places, but that will in no way make up for the fact that you're data will be burried deep inside the os, as opposed to freely available as it is in xml.
The progress of XPath
Pretty much anything that can be locked away in an os can be done better, and more flexibly in user space. That is why unix is better than vms, multics, or windows or whatever mainframe os, not because it has more features or higher speed, but becuase of it's light and flexible api. Files are stripped bare of anything more than the bare minimum. That keeps things flexible and easy, everything else is moved into the library.
You might want to look at this a little more in depth.
There are some major issue with this as a general file system. Orcale seems to want to push users interaction with the server/file system via a web client. If you use the SMB share or NFS share, you loose most if not all of the advance feature like versioning, check-in/out, etc. And don't even think about using the NFS extensions as they don't even support file locking!
Also, take your current storage space and triple or quadruple it to get the same effective space with the overhead induced by storing it in Orcale. (It took our company several weeks to get this rule of thumb out of Orcale as it is not in any of the docs.)