Slashdot Mirror

Isn't it true that much of Reiser4 is already coded? Since Reiser is GPL 2, there is nothing to stop the remaining developers -- or anyone else -- from picking up the ball and running with it. Namesys hired at least 1 or 2 full-time developers other than Reiser, although my guess is that since their Web site is currently down, there is probably nothing left of Namesys since Namesys is/was mostly Reiser, his (still missing, possibly deceased) wife Nina, and Reiser's father.

So, probably with no money, my guess is the hired hands won't continue work, but I am unable to speak for them.

Still, ReiserFS could continue without Hans, right?

Take a look at some contribution statistics for open source projects; the vast majority are one or two person shows, with maybe a handful of patches over vast stretches of time. That picture improves a bit for bigger projects, but ReiserFS doesn't really count as a big project. You'll of course appreciate this interview comment:

"I do not think that just being arrested will affect anything so long as Hans is not actually convicted," says Oleg Drokin, the former release manager at Namesys. "If he is convicted, that might cause problems for Namesys [because] it is operated solely by Hans."
The main concern, according to Drokin, is whether Namesys employees, many of whom live in Russia, will continue to receive their salaries. If the money stops flowing, "some people will stop working, of course."
However, Drokin thinks that situation is unlikely. "Hans suspected that he would be suspected from the very beginning," he adds. "I would think he took necessary steps for Namesys employees to continue to work even in his absence and even Namesys itself is safe for at least some time." -- http://www.linux.com/feature/57759, bold emphasis mine.

And sure enough, http://www.namesys.com is gone, and here's the article marking their demise: http://www.news.com/8301-13580_3-9851703-39.html. So yes, the code is still out there... just waiting for some developer to take a look at it, shrug, and demand that the whole thing be rewritten from scratch, because no one ever wants to take on an existing project and finish it off.

It's dead, Jim. The source code is out there, but without anyone to work on it, it might as well be a collection of ads for whalebone corsets. Of course, that's just my opinion, and if I'm wrong, so much the better for Linux.

The whole discussion above clearly shows a domain when modern operating systems have failed: information management. Living the definition of information to the applications creates the kinds of problems discussed above.

The solution is the following: Operating systems should define a mechanism for defining information types, including conceptual and physical details.

In other words, O/Ses should provide, by default, not a filesystem, but a database. If Unix came with a database by default, no such problems like the above would exist.

Reasons:

1- Reiser4 has fast journaling, which means that you don't spend your life waiting for fsck every time your laptop battery dies, or the UPS for your mission critical server gets its batteries disconnected accidentally.

2- Reiser4 is an atomic filesystem, which means that your filesystem operations either entirely occur, or they entirely don't, and they don't corrupt due to half occuring. Reiser4 does this without significant performance losses.

3- Reiser4 uses dancing trees, which obsoletes the balanced tree algorithms used in databases. This makes Reiser4 more space efficient than other filesystems, because it squishes small files together rather than wasting space due to block alignment like M$ does. It also means that Reiser4 scales better than any other filesystem.

Do you want a million files in a directory, and want to create them fast? No problem.

4- Reiser4 is based on plugins, which means that it will attract many outside contributors, and you'll be able to upgrade to their innovations without reformatting your disk. If you like to code, you'll really like plugins....

5- Reiser4 has a commitment to opening up the FS design to contributions. It is now adding plug-ins so that you can create your own types of directories and files.

6- Reiser4 is architected for military grade security. It is easy to audit the code, and assertions guard the entrance to every function.

For further details, and further technical knowledge, go to http://www.namesys.com/

open last dir/patch at ftp://ftp.namesys.com/pub/reiser4-for-2.6
file znode.c, item 5:

diff -puN /dev/null fs/reiser4/znode.c
--- /dev/null Thu Apr 11 07:25:15 2002
+++ 25-akpm/fs/reiser4/znode.c Wed Mar 30 14:55:08 2005
@@ -0,0 +1,1141 @@ /* Copyright 2001, 2002, 2003 by Hans Reiser, licensing governed by
* reiser4/README */ /* Znode manipulation functions. */ /* Znode is the in-memory header for a tree node. It is stored
separately from the node itself so that it does not get written to
disk. In this respect znode is like buffer head or page head. We
also use znodes for additional reiser4 specific purposes:

. they are organized into tree structure which is a part of whole
reiser4 tree.
. they are used to implement node grained locking
. they are used to keep additional state associated with a
node
. they contain links to lists used by the transaction manager

Znode is attached to some variable "block number" which is instance of
fs/reiser4/tree.h:reiser4_block_nr type. Znode can exist without
appropriate node being actually loaded in memory. Existence of znode itself
is regulated by reference count (->x_count) in it. Each time thread
acquires reference to znode through call to zget(), ->x_count is
incremented and decremented on call to zput(). Data (content of node) are
brought in memory through call to zload(), which also increments ->d_count
reference counter. zload can block waiting on IO. Call to zrelse()
decreases this counter. Also, ->c_count keeps track of number of child
znodes and prevents parent znode from being recycled until all of its
children are. ->c_count is decremented whenever child goes out of existence
(being actually recycled in zdestroy()) which can be some time after last
reference to this child dies if we support some form of LRU cache for
znodes.

*/ /* EVERY ZNODE'S STORY

1. His infancy.

Once upon a time, the znode was born deep inside of zget() by call to
zalloc(). At the return from zget() znode had:

. reference counter (x_count) of 1
. assigned block number, marked as used in bitmap
. pointer to parent znode. Root znode parent pointer points
to its father: "fake" znode. This, in turn, has NULL parent pointer.
. hash table linkage
. no data loaded from disk
. no node plugin
. no sibling linkage

2. His childhood

Each node is either brought into memory as a result of tree traversal, or
created afresh, creation of the root being a special case of the latter. In
either case it's inserted into sibling list. This will typically require
some ancillary tree traversing, but ultimately both sibling pointers will
exist and JNODE_LEFT_CONNECTED and JNODE_RIGHT_CONNECTED will be true in
zjnode.state.

How? The code is all open source. Hans owned the company called "Namesys" that does all the development, apparently that company is still alive to some degree anyways, because developers continue to post patches from @namesys.com addresses. All I know is that there still seems to be a decent amount of activity surrounding it (note the date, April 25th 2007).

My understanding is that Hans was the "brains" behind the operation as far as high level ideas and direction is concerned, but he spent a good portion of his time looking for funding and trying to raise money in other capacities, while he paid other programmers to work on Reiser4 itself.

As far as your comment regarding journaling and htree's, you are obviously severely lacking in knowledge as far as what Reiser4 is capable of, it goes FAR beyond fast access to many files and journaling. Speed, encryption and compression are just a few of the other major features it offers... According to the benchmarks, with the speed of CPUs today you can almost double your disk performance by enabling compression, contrary to what most people would think.

Reiser4 introduced us to all sorts of interesting capabilities never before seen in a file system (at the time) but I believe this same "layering violation" attitude pretty much put a stop to any of it getting into the kernel. The Reiser guys were forced to pretty much cripple their file system feature wise if they were to have any hope of getting it included in the kernel.

See Reiser4 Pseudo Files as one example.

I can understand that in certain cases "layering violations" are bad, but Linux kernel developers don't even seem to be willing to experiment or think outside the box at all.

Both sides have valid arguments... I don't think there is any easy solution, but it would be nice to see more forward thinking in the community.

For example I've got a shortcut to a file on C:\, which is 391 bytes actual size, 4096 bytes on the disk.

Hopefully, the filesystems, which can pack multiple short files into a block, will become more popular... ReiserFS does this, supposedly — I wonder, if anything else does...

Hans vision (http://www.namesys.com/whitepaper.html) is about unifying namespaces (fs, database, email, config files, etc.). So I believe his primary objective was to build a file system that performs well under all situations especially those situations where current filesystems perform so bad nobody writes software to use the fs that way. Most notably lots of tiny files.
Ext(2/3) use a traditional UNIX FS design and most software has been written to work acceptably on this type of file systems. In other words, nobody writes software that creates thousands to millions of tiny files because ext absolutely sucks at this (and ext4 doesn't help much) . So to get around this most of today's software create a new namespace in a single file. /etc is exactly this.
reiser(fs/4) also use extents which improves its handling of really large files which is becoming more common (movies, music, etc.). ext4 introduces this feature.
So for today's common usage patterns and file sizes ext4 will perform reasonably well compared to reiser4.

But if you where to extend fs semantics enough to make it reasonable to use the fs to directly back something like gconf (http://en.wikipedia.org/wiki/GConf), reiser4 would be substantially (10x) faster than ext(2,3,4) could ever hope to be. In principal if you extend the filesystem to directly support gconf and similar you would be able use of many fs tools for free.

Consider the windows registry. It is a mistake because it obscures the information by its convoluted structure and just as significantly, one can't use all those usefull tools on the registry you use with your file system (well not to many in Windows). Gconf addresses the first problem but not the second. Overall, I would consider the /etc approach superior because it is better documented, more transparent and I can use cp and subversion on it. Nobody will ever write a tool as powerful as subversion for the windows registry because the value isn't great enough. But subversion was never designed specifically to help with /etc it was primarily for coding but because coding has much in common with /etc the same tool can be used for both.

Yet, the windows registry had the good idea of creating a consistent way to store the the same type of data. Saving programming time and building a single consistent source for all things configuration. I believe it is possible to build something that gets the best of /etc and the best of the windows registry or better yet gconf.

The more use one can get out of a tool, the more people will work to develop and improve it. By using the same namespace for as much as possible, the more value a tool designed to work with that namespace will have. In other words, the utility of a computer system is proportional to the number of ways the components can interact, not the number of components.

To achieve this objective, one needs a solid foundation, a system that can efficiently store whatever you ask of it, that is what reiser4 is trying to be. But, clearly much work remains.

I've been to the namesys website and haven't found anything about the Hans Reiser arrest. If it is more than a one man show, then they should have a prominent statement about their intention to continue development regardless of the outcome. Not seeing something to that effect after this mych time would make me quite nervous if I had a business or product that relied on continued development of the ReiserFS line. Seems as if Namesys is accepting the inevitable demise of the whole organization at this point; I hope that changes.

http://www.namesys.com/support.html

"$25 gets you an answer to a question about anything."

Heck why not pony up the $15k reward for information, send it through that cgi, and just ask Hans: "Did you kill your wife?"

"If you don't get a solution that satisfies you, and you haven't asked us to do more than $25 of work, you get your money back. Just ask Reiser@namesys.com If it is more than $25, then refunds are at Hans Reiser's discretion, who claims to be reasonable."

Case solved, move along http://www.namesys.com/support.html

$25 gets you an answer to a question about anything.

Yeah, but they were dancing trees...

ReiserFS 4 can be used as a configuration database. And better, you can edit it with vi. Look here: http://www.namesys.com/v4/v4.html#aggr_files

Highly motivated people can often not devote as much time as they would like to OSS because they have to go to a regular job to pay for food etc.

There are a lot of key Linux developers who provide huge benefit to the community, but would like to make it pay so that they can make a fulltime job of it. Go look at what some people like Hans Reiser have to say http://kerneltrap.org/node/5654 "Doing GPL work is doing charity work in our current legal and economic framework. That should be and could be changed, but for now it is so. I have done my share of charity, and I would not have a problem doing proprietary work.", and http://www.namesys.com/ "For free software based on support revenues to be viable, people have to be more inclined to use our support service than they are to use the support services of persons who bundle our software with what they sell. Frankly, they are not, and this is why providing service on free software is failing as a business model for producing free software."

For my own part, I write OSS that saves people literally millions of dollars per year, yet I can only treat it as a hobby because it can't pay my bills.

Hopefully at some stage people start **paying** for stuff that is valuable to them. Unfortunately people grab what they can get for free.

Having good roads is very valuable, and you would not have those if they were not paid for. They are typically paid for by taxes because most people would not voluntarily dip into their pockets to pay for roads etc.

I think any methods that help get money into the hands of **key** OSS developers is a good thing.

how about ReiserFS, ghostscript, and our favorite lightning rod for flamefests MySQL? I was actually looking at namesys's and an average joe can even contribute code to the project as long as every submition contains an legal agreement which I assume allows them to dual license.

"... we choose to give our software away for free for use with software that is given away for free (e.g. Gnu/Linux). Since we don't have a lot of illusions about our ability to entirely change the world, and it is amusing to sell free software, for those who do not want to disclose their software and do not want to give it away for free, we charge a license fee and let them keep their improvements to our software without sharing them. These fees help substantially in allowing us to survive as an organization." www.namesys.com

ReiserFS? Check out the "support page" for it: http://www.namesys.com/

Pay $25 if you need support... And apparently, judging by the number of filesystem corruption problems reported with Reiser and the 2.6 kernel, you probably need a lot of support. I've been bitten by ReiserFS corruption problems in the past year on a dev system. Maybe it's gotten better, but six months since the last serious problem was diagnosed is too short a time for my comfort level.

After 10 years working to create a suitable storage layer for implementing semi-structured data queries in the FS for Linux, I gotta tell you, this stuff is harder than I ever thought it would be. :-/

(See Reiser4 for our storage layer, and our Future Vision paper for what semantics we are going to add.)

5 years to do the first draft (ReiserFS V3), and then another 5 years to get it finally right (Reiser4).

To do enhanced semantics cleanly, you want keywords to be just another kind of file (see our Future Vision paper for why. That means you need to store files that contain phone number sized objects and keywords reasonably efficiently. Because of network effect economics creating a barrier to entry, you have to at the same time make traditional file system usage patterns at least as fast. That is hard. How hard? Oracle tried to do it without deeply changing their tree algorithms, amd implemented an FS on top of their database engine, and found that it was half the speed of a traditional filesystem. Others also found it hard. I tried to do it with V3, and found that for files in the 0-10k size range, I had many of the performance problems that FFS had when they created fragments. Thing was, I never knew they had performance problems, because it was not in their paper.... The problem was that when you combined fragments from multiple files, you add seeks, and one added seek is deadly to performance. The approach used in most databases, BLOBS, suffers from the same problem as FFS combining fragments, and yet more, because BLOBs unbalance the tree (see our website for details and nice diagrams). The usual transaction technology employed for databases, it is just wrong for filesystems, what you need in an FS is to fuse multiple transactions together into batches. And more....

There are so many different areas where if you take a wrong step, performance goes through the floor. You cannot imagine how depressing it is to work on a project where the performance is terrible until the very end, after 5% to rarely 20% at a time you've dragged it into something decent over years of time. I look back on it, and I see that we were incredibly lucky, because all the mistakes I made, were mistakes that took days or weeks to fix, and except for one thing (BLOBs), all the major things that would take years to fix, I got right. There is no reason for this other than luck. And BLOBS cost us years.

So we have for Linux the storage layer that MS could not develop because they quit before 10 years had passed, and perhaps weren't lucky enough at. Now, with technology working, and balance trees that can emulate file system semantics at twice the speed of the real thing (see our benchmarks ), sigh, if only we can overcome the politics. Yup, the WinFS team had to deal with corporate managers that quit before 10 years are past, but we have to deal with..... better unfinished as a sentence.

The only consolation in this field is that everyone else seems to find it just as hard. Probably that includes even the politics.

After 10 years working to create a suitable storage layer for implementing semi-structured data queries in the FS for Linux, I gotta tell you, this stuff is harder than I ever thought it would be. :-/

(See Reiser4 for our storage layer, and our Future Vision paper for what semantics we are going to add.)

5 years to do the first draft (ReiserFS V3), and then another 5 years to get it finally right (Reiser4).

To do enhanced semantics cleanly, you want keywords to be just another kind of file (see our Future Vision paper for why. That means you need to store files that contain phone number sized objects and keywords reasonably efficiently. Because of network effect economics creating a barrier to entry, you have to at the same time make traditional file system usage patterns at least as fast. That is hard. How hard? Oracle tried to do it without deeply changing their tree algorithms, amd implemented an FS on top of their database engine, and found that it was half the speed of a traditional filesystem. Others also found it hard. I tried to do it with V3, and found that for files in the 0-10k size range, I had many of the performance problems that FFS had when they created fragments. Thing was, I never knew they had performance problems, because it was not in their paper.... The problem was that when you combined fragments from multiple files, you add seeks, and one added seek is deadly to performance. The approach used in most databases, BLOBS, suffers from the same problem as FFS combining fragments, and yet more, because BLOBs unbalance the tree (see our website for details and nice diagrams). The usual transaction technology employed for databases, it is just wrong for filesystems, what you need in an FS is to fuse multiple transactions together into batches. And more....

There are so many different areas where if you take a wrong step, performance goes through the floor. You cannot imagine how depressing it is to work on a project where the performance is terrible until the very end, after 5% to rarely 20% at a time you've dragged it into something decent over years of time. I look back on it, and I see that we were incredibly lucky, because all the mistakes I made, were mistakes that took days or weeks to fix, and except for one thing (BLOBs), all the major things that would take years to fix, I got right. There is no reason for this other than luck. And BLOBS cost us years.

So we have for Linux the storage layer that MS could not develop because they quit before 10 years had passed, and perhaps weren't lucky enough at. Now, with technology working, and balance trees that can emulate file system semantics at twice the speed of the real thing (see our benchmarks ), sigh, if only we can overcome the politics. Yup, the WinFS team had to deal with corporate managers that quit before 10 years are past, but we have to deal with..... better unfinished as a sentence.

The only consolation in this field is that everyone else seems to find it just as hard. Probably that includes even the politics.

After 10 years working to create a suitable storage layer for implementing semi-structured data queries in the FS for Linux, I gotta tell you, this stuff is harder than I ever thought it would be. :-/

(See Reiser4 for our storage layer, and our Future Vision paper for what semantics we are going to add.)

5 years to do the first draft (ReiserFS V3), and then another 5 years to get it finally right (Reiser4).

To do enhanced semantics cleanly, you want keywords to be just another kind of file (see our Future Vision paper for why. That means you need to store files that contain phone number sized objects and keywords reasonably efficiently. Because of network effect economics creating a barrier to entry, you have to at the same time make traditional file system usage patterns at least as fast. That is hard. How hard? Oracle tried to do it without deeply changing their tree algorithms, amd implemented an FS on top of their database engine, and found that it was half the speed of a traditional filesystem. Others also found it hard. I tried to do it with V3, and found that for files in the 0-10k size range, I had many of the performance problems that FFS had when they created fragments. Thing was, I never knew they had performance problems, because it was not in their paper.... The problem was that when you combined fragments from multiple files, you add seeks, and one added seek is deadly to performance. The approach used in most databases, BLOBS, suffers from the same problem as FFS combining fragments, and yet more, because BLOBs unbalance the tree (see our website for details and nice diagrams). The usual transaction technology employed for databases, it is just wrong for filesystems, what you need in an FS is to fuse multiple transactions together into batches. And more....

There are so many different areas where if you take a wrong step, performance goes through the floor. You cannot imagine how depressing it is to work on a project where the performance is terrible until the very end, after 5% to rarely 20% at a time you've dragged it into something decent over years of time. I look back on it, and I see that we were incredibly lucky, because all the mistakes I made, were mistakes that took days or weeks to fix, and except for one thing (BLOBs), all the major things that would take years to fix, I got right. There is no reason for this other than luck. And BLOBS cost us years.

So we have for Linux the storage layer that MS could not develop because they quit before 10 years had passed, and perhaps weren't lucky enough at. Now, with technology working, and balance trees that can emulate file system semantics at twice the speed of the real thing (see our benchmarks ), sigh, if only we can overcome the politics. Yup, the WinFS team had to deal with corporate managers that quit before 10 years are past, but we have to deal with..... better unfinished as a sentence.

The only consolation in this field is that everyone else seems to find it just as hard. Probably that includes even the politics.

Tagged as dukenukemforever. I could've told you this long ago...

Meanwhile, Hans Reiser is doing some real filesystem innovation...

the premise that Reiser is more stable than ext3 "because it has been out longer"

It's dishonest to put something in quotes when it's not a direct quote. The exact quote is:

"We don't touch the V3 code except to fix a bug, and as a result we don't get bug reports for the current mainstream kernel version. It shipped before the other journaling filesystems for Linux, and is the most stable of them as a result of having been out the longest. We must caution that just as Linux 2.6 is not yet as stable as Linux 2.4, it will also be some substantial time before V4 is as stable as V3."

There's a substantial difference between saying that something is more stable "as a result" of something and more stable "because" of something. He's not claiming that being out longer intrinsically makes it more stable as your misquote suggests, he's claiming that it led to reiserfs becoming more stable - because of the practices he mentioned.

In short - something being out longer == more stable? No. Something being exposed to lots of real-world use and receiving only bugfixes == more stable? Yes.

the quote from Adam Smith

He didn't quote Adam Smith, he drew an analogy between what he was saying and the network effect. It's an entirely reasonable analogy.

the ridicule of the unix approach of everything as a file

What ridicule? He's actually supporting that approach. For example:

Can we do everything that can be done with {files, directories, attributes, streams} using just {files, directories}? I say yes--if we make files and directories more powerful and flexible. I hope that by the end of reading this you will agree.

Would you care to point out where you thought he was ridiculing the UNIX approach?

all the naked people covered in newsprint

Yeah, they look dumb, don't they?

Anyone have a "more technical" link

I can only assume you mean something other than "technical".

without dancing trees

Dancing trees are a fundamental part of the design. How are you meant to understand the filesystem without understanding dancing trees?

and with a bit about how to recover your filesystem when something goes weird with the hardware even if the filesystem is perfect?

Ah, you don't mean technical at all, you mean practical for somebody who is entirely uninterested in the way the filesystem works. Perhaps Reiser4 Transaction Design Document is what you are after, but I doubt it.

Reiser4 does this.

"what they are trying to do hasn't really been done before"

The Hierarchical File System
DBFS
Project: OCFS

`next to ZERO file formats that are currently in widespread use by the computing world know anything about "metadata"'

ReiserFS
XFS

This announcement makes more sense than I think people recognize, particularly if you posit that they ran into performance difficulties.

Remember that they were going to put it all into the filesystem? Then they put it into a layer above the FS. Bet you a bitflip that they found performance problems resulting from mixing large files and small files in the same filesystem. It is very easy to have such problems, you have to get quite clever to avoid them, and expend a lot of effort on it. Rather than take the extra time, they pulled the enhanced semantics out of the filesystem. That was the wrong thing. Then, look at the descriptions of how some of the queries they supported took too long. It is really easy to design things in that area wrong, and get unacceptable performance impacts. Rather than solving the deeply challenging problems, they punted and put the stuff into SQL server. Why? Because people who don't want things to be slow can just not use the feature until they figure out how to make it not cost performance. Of course, that means no OS integration but..... it is so nice to not be designing Reiser4 by committee.

I see Microsoft responding to difficult technical problems not by solving them, but by running from them, and that explains the entire trajectory of WinFS.

Another consideration you can see between the lines is that they don't want to lose the revenue from SQL Server by doing everything that it does in the OS and doing it better. Marketers will do things like make the first release of something only available at a higher price. They do that a lot. They'll do it even if it robs Vista of most of its excitement to do it.

Large corporations often have real problems handling tough research projects.

Reiser4 took 5 years to get into working at all (v3), and 10 years of sustained development to get right (reiser4), and it is just the storage layer. You can't do that in a large corporation.

In a large corporation you are thinking that you need 3 years to do a project that is a paradigm change, and you go talk to management, and you sense that they have patience for 9-18 months, and you really want to do the project, so you tell them you can do it in 9-18 months.

18 months go by, and you are 1/2 of the way through the first version (you think you are 90% of the way through), and the first version is going to suck badly and take years to be well optimized. Now, if your product is the first in its market, you can make it even though it sucks, and get the money for the version 2. If you are going into a mature market, well, things are tough. Very tough. WinFS is going into a mature market.

Now, into this reality throw corporate managers. They think that if they intimidate the programmers a lot, products ship sooner. So, technical shortcuts get taken. Only problem is, in a product like WinFS, going into a mature market, taking technical shortcuts kills things. Especially since for a product like WinFS the technical shortcuts affect DEEP decisions that you will never be able to reverse out of. Like, whether the enhanced semantics are in the FS layer. Or whether the whole OS is designed around using the enhanced semantics in every component. Then, managers feel the need to prove they are tough about schedules, and they cancel for being late projects that everyone should have known were going to take a long long time because they were hard. There is some very interesting recent research suggesting that if you want an accurate project length forecast, you don't ask for an estimate, you create a betting pool.

The sad thing is, since everyone copies Microsoft, now there will be more people saying that Reiser4 shouldn't do what WinFS backed away from. We can do it. We solved the hard storage layer design problems, our stuff works. Now we can finally go after the enhanced semantics. It took 10 years, but we got the storage layer into the shape we want it in, and one plugin at a time the enha

It seems to me that you've started with your conclusion - that ReiserFS is not innovative - and used that to draw the conclusion that it must therefore suffer from the same problems as other filesystems. This is backwards logic. Please reverse your logic - look at its actual features and use that to decide whether or not it is innovative.

In particular, you seem focus on the need for application support. The whole philosophy behind ReiserFS is that separate namespaces for files and metadata is a bad idea, and that metadata should be exposed in the existing namespace - as files and directories.

The consequence of this is that applications don't need special support for ReiserFS's metadata. If, for example, you wrote an MP3 plugin that exposed the ID3 metadata, you would be able to search this with grep and locate, and edit it with vi and EMACS, without any changes whatsoever to grep, locate, vi or EMACS. Since the filesystem is exposing this metadata as files and directories, anything that supports files and directories will be able to access the metadata.

So the problem of metadata might mean that other filesystems require special application support, but it's precisely because ReiserFS is innovative that this isn't a problem for it. Which other filesystems are tackling metadata in this way? Are there any?

There's two ReiserFSs for the purposes of this discussion.

There's ReiserFS 3, which pretty much anybody running ReiserFS right now is probably actually using. It is, as near as I can tell, basically as you say a POSIX-style filesystem that has some strengths and some weaknesses. It's good with small files under certain circumstances, and is generally somewhat more efficient with space. (Most of its other unique-at-the-time features have since been copied by the other open source FSs, and matched by the open-sourced commercial FSs.)

Then there's ReiserFS 4, which is a nearly complete reconceptualization of how a file system works. You can read about it here. This is what people put up against WinFS, although ReiserFS is not itself a "relational" file system that I can see; I think it's more a case of trying to provide the same advanced functionality in two very different ways. It's interesting and I wish the ReiserFS 4 project well, but it's a damned hard problem and I don't know when or if it'll ever be done. (And they'll get all the more kudos from me if they pull it off.)

Some quick Google searching shows people claiming to use some form of ReiserFS 4 (since 2004 at least), but I don't know what the real status of the project is. The webpage doesn't seem well updated for that use. As near as I can tell my Gentoo-patched 2.6.15 kernel doesn't have 4 as an option, but I note the Portage does have a "reiser4progs" entry which claims to be 1.0.5.

If you're interested in learning more, that's where to start; be sure you're not reading about ReiserFS 3.

How about this white paper? Or the features already existing in Reiser 4? Doesn't sound like complaining at all, but as true innovation, which Microsoft is obviously lacking...

How about this white paper? Or the features already existing in Reiser 4? Doesn't sound like complaining at all, but as true innovation, which Microsoft is obviously lacking...

And meanwhile ReiserFS on Linux provides much of the functionality today that WinFS only promised for the future.

I think that you have to inform better, Raiser has journaling. From namesys site: "Reiser4 is an atomic filesystem, which means that your filesystem operations either entirely occur, or they entirely don't, and they don't corrupt due to half occuring. We do this without significant performance losses, because we invented algorithms to do it without copying the data twice"

On Linux, the ReiserFS and Reiser4 filesystems "squish small files together" to avoid this problem.

Hans Reiser's Namesys already does that. Check out reiserfs for something tried and true, or check out reiser4 for a new outlook on filesystems and better performance.

But yeah, I've been feeling for a long time now that programming and system work is needlessly tedious, boring, anal retentive, and that it's the fault of the languages and systems. We hope and strive for but don't yet expect intelligence from our computers, just very very fast computation. Intelligence is a fine goal, yet we haven't gotten computers entirely set up to relieve us of what they can do well. Somehow it's people who often end up having to do a lot of mindless tedious repetitive tasks for the computer. Shouldn't installing an OS be a nearly brainless task? Shouldn't configuring the Linux kernel be automatic? Why, if I want to "roll" my own kernel, do I have to dig through hardware detection messages from system boot, from ls utilities like lspci, and so on to figure out how to configure the kernel? "Cobolization", that's one of the problems. Ever see source code for "hello world" in Xlib? Over 100 lines of code just to put up a lousy little message window. (Of course that's why we have Motif, GNOME, KDE, etc.) XML isn't much more "human readable" than assembler. Why do we have a separate language for makefiles? I don't know a single general purpose language that isn't universal, but somehow none of them are suitable for instructing the computer in the details of compilation. As if makefile language isn't bad enough, C has 2 syntaxes for the same concepts: preprocessor syntax such as "#ifdef #endif" for code that should be compiled into a program if a condition is satisfied, and "if () { }" for the exact same thing except executed during run time instead of compiled during compile time. More fundamental than file/memory access, we have dozens of incompatible and arbitrary syntaxes for calling functions and passing parameters. Shells do it one way, most apps parse command line options in 1 of 2 ways, but many mix it up, and C does it at least 2 ways in syntax: "main(argc, argv) int argc; char **argv; {" or "main(int argc, char **argv) {", and 2 ways internally-- who remembers that C has a keyword "Pascal" and what it does? Thanks to so much of most OSes being written in C, most other languages have their own special conventions for calling C library functions. As to arguments that each of these languages is a tool, that tools are not universally suitable for every application, and one should use the best tool for the particular purpose of the moment, note that this is not true of the typical natural language. I do not need to switch from English to another language to express ideas. Any universal language can express what any other can with some constant multiple of the number of statements used in either one. For natural languages, that constant is a lot closer to 1 than for programming languages. Not one of these programming languages approaches or intends to approach the power and expressiveness of a typical natural language. Well, that would take machine intelligence, but there is much that can be done to streamline and simplify what we do have now.

That's not saying that Linus hasn't been the main driving force all these years, but he's in no way the only one. Now, the jury's still out as to whether GPLv3 is a good thing - I haven't made up my own mind yet either - but I'll bet large chunks of the Linux kernel could be relicensed immediately if the will was there.

Since iRobot has strong MIT roots and congratulated the first hackers who modified their Roomba. Later, iRobot released full specifications for the original Roomba and more hacking efforts bloomed.

Oh, and as for the government? This is the same government which released BRL-CAD and NASA World Wind, and sponsored the development of the Reiser filesystem and OK WebServer (kicks Apache in the ass for dynamic sites) among countless other open source projects. Oh, and heard of SELinux? From the big bad NSA? This is all irrelevant though; the DoD did not design this vehicle, iRobot did. iRobot owns the code, not the government

So maybe building a full blown distro that competes against Redhat and Suse is unlikely... but I could definitly see Google making major feature enhancements in the linux desktop search realm.

In short, I'm convinced the registry doesn't require a separate implementation from the filesystem.

Designers (including Mozilla's) are entrenched in the idea that lots of tiny files are bad. Traditional filesystems and even api's to some extent aren't optimized for that. But Microsoft was in a different position, because the designers of the registry were in cahoots with the filesystem people (same company). Instead of inventing the registry, they should have optimized NTFS for config info.

In short, I'm convinced the registry doesn't require a separate implementation from the filesystem.

Designers (including Mozilla's) are entrenched in the idea that lots of tiny files are bad. Traditional filesystems and even api's to some extent aren't optimized for that. But Microsoft was in a different position, because the designers of the registry were in cahoots with the filesystem people (same company). Instead of inventing the registry, they should have optimized NTFS for config info.

What you call "open source" programs are just some loosely assembled code lines stolen from companies that make real software.

Real software? You mean, like Real Player? No loosely assembled code lines for sure.

Do you think a bunch of hippies would do something useful, apart smoking their pot.

At least I'm doing something useful right now! Thanks for the support, I'll keep up the good wo.. err.. smokin' :)

No, they write software that is outmoded since 1983, and call thelm "free".

Mhh, there's a misunderstanding here sir. Open source may be what you describe, but free (as in speech) software isn't at all. It isn't about code, it's about the software liberty, and in my experience, the code is often better than proprietary code (which nobody sees so there's nothing to be ashamed of, maybe FOSS haters are just jalous that some people can actually produce code that can be shown? :)). Some would say: Software libre takes more than a license, it takes a design.

Isn't this exactly the kind of security that Reiser4 wants to address via its plugins system, while also adding more speed to the filesystems?

ZFS - the best filesystem ever devised.
Is ZFS better then Reiser4?

The article probably misses the point by looking at files of omogeneous type, but the problem with current... erm... paradigm (sorry for the swearword, couldn't think of a better one) for managing files is real. The problem lies not in the manager, but in the storage mechanism itself: the hierarchical filesystem. We all hit the point, sooner or later, where the organization of data in our mind can't be matched with a rigid directory hierarchy. What is really needed is a graph organization. Links and metadata help up to a point, but in order to make things work smoothly it will probably be necessary to abandon current ideas about directories as "containers", slash-separated file paths, filenames as unique identifiers (and attributes as mere extensions of the filenames), probably even the graphical interface used to access and manipulate the filesystem. The listed applications do their best, but they still work at a level too high to replace a file manager. I don't have a solution. Maybe Reiser4 can be of some help.
Oh, I guess this will be modded down because I said "paradigm", anyway...

I agree to some extent.

But... A database is a file system, and a filesystem is a database.
You're in fact opposing hierarchical data organization to relational data organization.

I suppose Reiser4 is the way to go...

"It has to be tested for compatibility, quality and performance. You can't be losing data, and if it doesn't offer a performance benefit over UFS or UFS2 then there's very little point in porting it."

Clearly you don't know about Reiser (no offense, it's just that that question shows a stark lack of understanding with regards to why Reiser is interesting in the first place).

Reiser solves one of the oldest problems facing the old Unix-style filesystem: the adoption of btree-order performance directory lookups (using Reiser's "dancing trees") without significant loss in other areas of filesystem performance, e.g. directory entry creation and deletion, etc. This is something which was long thought impossible.

This lead to further development, since the major reason to avoid creating thousands of temporary files has always been directory lookup times. So, now the question is: how far do you go with files? Reiser 4 answers that question by adding significant semantics to files which were not practical with slower filesystems (again, keep in mind that when I say "slower" I refer to the performance bottleneck surrounding large directories primarily).

The problem with Reiser is that it is Reiser, and none of the exisiting filesystems can match its performance in these areas. That means that if you write an application that relies on Reiser's performance, it really RELIES on Reiser, and cannot perform well under normal filesystems without significant engineering (e.g. writing special-case code for Reiser and non-Reiser filesystems). In some cases (e.g. databases) this might be worthwhile, but in the case of more mundane applications, having filesystem-specific code is not always viable.

For more information, see the Reiser4 site: http://www.namesys.com/v4/v4.html

Or set the company up to succeed because you're dead anyway...like OS X. I don't understand why anyone could be so adamant about maintaining codebase when a great many products are grown to be ugly functional messes. Windows XP SPECIFICALLY. Rewrites are good, this is what factoring is about. Refactoring design is GOOD as technology progresses. There's no real argument against it (other than fear and complacency). This is particularly applicable to the philosophy of the http://www.namesys.com/v4/v4.html Reiser4 filesystem design. It didn't come about because it relies on assumptions in Ext2, but because it challenged and tossed them. Statistically staying with the status quo is good, but that's because most managers are technologically inept and surprisingly good at writing reports and marketing fliers.

While not a donations page per se, you can visit their support page: http://www.namesys.com/support.html where they accept credit card payments for technical support. I'm sure that they wouldn't mind if you dropped some money in there and told them that it's a donation rather than a support request.

``How about the semantics described in The ReiserFS Future Vision Whitepaper ;-)''

Looks very good. I've not read the whole paper (it's a bit longish for just figuring out the syntax and semantix of queries), but here's how I understand it (and do correct me if I'm wrong):

  - A query is a list of things to match
  - These things can be simple keywords, ordered keywords, pruners, or subqueries (groups).

So a simple search would be something like [book store Miami], which would find anything somehow related to book and store and Miami; most likely book stores in Miami. If you wanted to make that explicit, you could say something like [type/store selling/books location/Miami] (I've avoided the use of pruners; they seem like a less general form of ordered keywords). And if you want a store in Miami that sells books by Tolkien you could try [store Miami [books by Tolkien]]. Am I on track so far?

I think the key point you're making is that any sort of organization can be helpful in some circumstances, but create barriers in others. Therefore, the only structure you can safely impose without a-priori knowledge of the data and how it will be used is no structure. This is very true, as anybody who has maintained databases should know (or probably anybody who has organized data).

Now, here are a few problems that I haven't seen addressed:

1. What if I want to have unequal relations? A store selling books older than 1960? Or a store selling books which are not The Lord of the Rings?

2. You mention "Custom programmed syntax" (which is a great feature to have), but no mechanism to actually introduce this. Also, does such syntax live within the same constraints your syntax already imposes (like Lisp macros), or can you freely extend the syntax in any way imaginable?

3. How does one create those queries programmatically? The problem with most other systems is that the syntax of the query language is sufficiently different from that of the programming language that the only mechanism flexible enough to compose queries is string concatenation. Unfortunately, that also opens the door to injection vulnerabilities. Practice shows that programmers are not smart enough to consistently escape user-supplied data, or to create wrappers that do so automatically. Do you have a plan to address this problem?

``Regarding how to compute it, the standard algorithm traditional to boolean algebra works reasonably well, in which you take the intersection of all of the sets matching the subnames, starting the computation with the smallest of those sets.''

Yes, of course. My question about the computation was more related to where it happens: in the database, or in the program querying the database. Your paper puts all the computation in the database, which I think is the right approach, especially when you can extend what the database can do (stored procedures in SQL, custom syntax in your case). The tradeoff here is complexity of the database and query language vs. efficiency and duplication of work. I think reiser4's plugins are a great way to sidestep this problem; the database is simple at the core, but can be extended to match pretty much any degree of complexity. But again, the question arises: can you load these plugins through the query interface (as an unprivileged operation), or does it have to be done externally (and possibly as a privileged operation), like Linux kernel modules?

All these questions severely impact the flexibility and safety of the system, and I'm afraid there's no one size fits all solution.

How about the semantics described in The ReiserFS Future Vision Whitepaper ;-)

Regarding how to compute it, the standard algorithm traditional to boolean algebra works reasonably well, in which you take the intersection of all of the sets matching the subnames, starting the computation with the smallest of those sets.

Hans