Slashdot Mirror

GPFS has a different focus, but XFS seems to be aimed at solving similar problems as ReiserFS (scalability, high performance, journaling).

If benchmarks are even halfway legit, then this is indeed something amazing.

ReiserFS has a 16TB limit on 4KB blocks.
However, its ability to pack the tails of the files in single blocks reduces even more space waste.

What? Anyone sayn' it's not for Windows?
Who cares?

why limit your self to NTFS? The file system I am currently using can handle 16TByte . Besides Reiserfs there are many other filesystems. And I am sure that most of them do not have those small limits NTFS has.

you might be interested in link hans reiser posted above in this article.
http://www.namesys.com/whitepaper.html
i just started reading it, but there is no way i could read it in one go.

Reiser4 is technologically ahead of WinFS as a high performance storage layer, see www.namesys.com for details on its design. When you do this layering the way they did it, with the metadata stored in a layer above the FS rather than integrated into it, you lose a lot of performance while gain the advantage of successfully avoiding dealing with a host of technical issues. We are at least 5 years ahead of them technically in the storage layer.

That said, semantic enhancements matter more than performance, and it is better to do something semantically than to do nothing, and what Linux currently is doing is nothing.

The political support for adding semantic enhancements to Linux namespaces is mixed at best. I worry we will see that death by committee rules, and there will be no belief that each FS should try to innovate in its own way and compete with the others until one is proven the right solution. We are in serious danger of having MS implement bad technology, and Linux having to devote large amounts of resources to copying it in 5 years because we were late and chose to trail rather than lead. If the filesystems were free to compete in semantics, we could have one or several of the Linux filesystems leading them instead.

SQL and the relational model is fundamentally the wrong model for semi-structured data. See www.namesys.com/whitepaper.html for why.

Technically, I would worry much more about Apple. Dominic Giampaolo is very bright, and well funded. His chances of delivering on a good set of semantics are high because he and Jobs are very sharp, and neither of them is afraid to go where no one has gone before. Our chances of losing technically to Giampaolo and Jobs are high, because we are frankly not well funded, and a lot of us are complacent with semantics that are still pretty much the same as their father's Unix box.

So, in summary, I would say that we are still ahead but losing speed fast.

Thanks for your kind words Hisham.

Reiser4 is technologically ahead of WinFS as a high performance storage layer, see www.namesys.com for details on its design. When you do this layering the way they did it, with the metadata stored in a layer above the FS rather than integrated into it, you lose a lot of performance while gain the advantage of successfully avoiding dealing with a host of technical issues. We are at least 5 years ahead of them technically in the storage layer.

That said, semantic enhancements matter more than performance, and it is better to do something semantically than to do nothing, and what Linux currently is doing is nothing.

The political support for adding semantic enhancements to Linux namespaces is mixed at best. I worry we will see that death by committee rules, and there will be no belief that each FS should try to innovate in its own way and compete with the others until one is proven the right solution. We are in serious danger of having MS implement bad technology, and Linux having to devote large amounts of resources to copying it in 5 years because we were late and chose to trail rather than lead. If the filesystems were free to compete in semantics, we could have one or several of the Linux filesystems leading them instead.

SQL and the relational model is fundamentally the wrong model for semi-structured data. See www.namesys.com/whitepaper.html for why.

Technically, I would worry much more about Apple. Dominic Giampaolo is very bright, and well funded. His chances of delivering on a good set of semantics are high because he and Jobs are very sharp, and neither of them is afraid to go where no one has gone before. Our chances of losing technically to Giampaolo and Jobs are high, because we are frankly not well funded, and a lot of us are complacent with semantics that are still pretty much the same as their father's Unix box.

So, in summary, I would say that we are still ahead but losing speed fast.

Thanks for your kind words Hisham.

"I am having extensive problems using ReiserFS; it seems to have bugs all over the place. I'm not compiling with a buggy compiler. What is happening? How can this be stable?
You have hardware problems. Really, you do. Even if the bugs don't show up with ext2, you have hardware problems. (See FAQ question about ReiserFS running 3C hotter than ext2.) Most SuSE users use ReiserFS. Obscure bugs probably still exist; but if you find bugs as easily as using Windows, you have bad RAM, bad CPU, bad cable, bad cooling, VIA chipset with PCI quirks turned on, or other hardware or other software layer bugs. ReiserFS is stable. You can be sure that if the bugs are encountered easily and commonly with normal usage patterns, it is not us. This does not mean that the next release won't somehow break something though :-/..... Real bug reports are at the time of writing outnumbered 10 to 1 by hardware bugs that trigger error messages. We are working on making our error messages better at catching hardware bugs and identifying them as such. There is only so far we can go though in runtime consistency checking without serious speed reductions. We don't release software unless it goes through extensive testing; so if you don't think that our testing could have missed the bug, it is probably hardware."

Referenced in the FAQ.

Naw. I'd say that's BSD ;)

mean, why is it that everyone is getting so 'uptight' here about that anyhow? I don't see Linux with a DB driven filesystem either!

Honestly, I don't think that DB is the way to do it either. I find indexing (ala Tenor/Spotlight) a much better solution. Regardless of that, though--you must admit that the Windows search engine blows.

And, in a related topic: Most filesystems are, in fact, database driven. They use many of the same algorithms, provide atomic operations, and have queries (file locations). It just so happens that they don't use SQL to do it.

(Windows NT-based Os' are built to have an extensible filesystem)

May I be the first to plug Reiser 4?

However, it's obvious many here have never written code & certainly not of enterprise class size, because expecting to be able to do it in a heartbeat or miracles as others stated about doesn't happen overnight

Well, the expectation can happen overnight, but the programming certainly can't. ;)

Personally I think the current filesystem arrangement on Windows Server 2003 is just fine and it has been fine for ages. Windows Server 2003 is the core code of the next release, LongHorn, it's foundation. It is stable and solid as a rock imo. I have been using it for all of this year 2005 and much of 2004 as well. I can safely make that statement.

And you could say the same for HFS+, ext3, & reiser3. What's your point here?

However, again, the more I come to slashdot, the more it seems it is just ammo for the pro linux zealot's jihad against Microsoft with it not being in these Longhorn beta

Are you new here? I've been around for a few years now, and it's always looked this way, to me. ;)

Note: most of this made purely in jest :)

Well, I'm a bit disappointed that native Reiser4 support wasn't included in this release. It's one of the features I'm greatly looking forward to...and I'm too noobish to compile a Reiser4 kernel module myself.

It's not clearly advertised for some reason, but Hans makes available patches that add Reiser4 support to the mainline kernel. This should probably be spelled out more clearly somehwere, since I know alot of people avoid trying Reiser4 because they don't want to run -mm.

Sure, there are interesting efforts such as JNode, and people working on really cool new concepts like Zero-Install and Hans Reiser's vision of tomorrow's filesystem, but who is working to combine these and other concepts into something truly new and innovative?

"Enough stuffing metadata into filenames. Enough shoehorning all data into a file/folder/cabinet model, now less familiar to people than the networked infosystems that mimic them."

Well, that seems fine, except for the fact that you want your filesystems to PERFORM well. You think nothing of launching 10,000 "queries" at your filesystem, but when it comes to dealing with a relational database, you have to back off and think about caching strategies because the database is expensive.

Why? Because that flexibility requires abstraction, which costs cycles and memory.

Instead, why not re-design the filesystem to handle the most basic operations required by both databases and the infosystems that resemble, but are not databases... fortunately, someone has already done much of that for you.

How many times have we heard of huge sites going down because databases become corrupt or unrecoverable, or of the huge resource strain (memory and CPU) from a large database?

In my opinion, the future of databases is nothing so complicated as pitched here -- but rather a move to simpler, more reliable back ends where the filesystem is the database. This is certainly the vision pitched by Hans Reiser and reiserfs, which aims to put more database like intelligence within the filesystem. So you eliminate extra unnecessary layers that just eat up resources and create fragile databases.

Well, it looks like even some filesystems that don't suffer much from fragmentation also have an automatic defragger such as Reiser4 with something they call the repacker.

Well, it looks like even some filesystems that don't suffer much from fragmentation also have an automatic defragger such as Reiser4 with something they call the repacker.

Jim Allchin details various planned Longhorn features to meet this goal, such as auto-defragmenting in the background

Here's something that works: implementing a file system that doesn't require constant defragmentation.

If Clippy is popular then why did Microsoft decide to hide it by default?

With regard to the registry: it violates two principles of good software architecture. First, you want to minimize the multiplication of namespaces and data access methods. Second, you don't want corruption of a single file to result in an inability to access your whole system.

You could argue that relational databases tend to violate these principles as well but the truth is that when they are used in volume applications they _are_ the data store and the metadata store and the single, unified namespace as well. They also have many more corruption-fighting features than does the registry.

Something like ReiserFS's plug-ins might be useful here. See this example of how /etc/passwd could be split up to improve privilege granularity (search for "Take /etc/passwd for example"), all without breaking existing programs. This would also give you finer-grained modification timestamps, which would help you figure out what that pesky installer just did.

Although we use MySQL's transactional InnoDB tables, they can still sometimes be left in an unrecoverable state

It would be simple enough to write a plugin for Reiser4 FS which would do what you want. Why not give it a try?

I like the ideas I see on ReiserFS future vision. A lot of posts here mention problems with the whole idea of rwx permission bits, or problems with tracking of installs. Applications get scattered all over the directory structure, or compromises are made to prevent that. I thought that ditching the whole idea of a directory hierarchy was the way to go. Reiser FS thinks so too. No more directory trees!

An example to illustrate the problem. Suppose there are 2 apps, "A" and "X11", and each one is composed of many executable and help files. With directory trees, we only have a choice of which mess to make. Throw everything into /usr/bin and /usr/man or create /usr/man/man1/A/, /usr/man/man1/X11/, /usr/bin/A/, /usr/bin/X11/, and maintain package data with yast, apt-get, rpm, pkgtools, or whatever? Or create /usr/A/man, /usr/A/bin/, /usr/X11/man. /usr/X11/bin/ and so on, keeping everything to do with A under /usr/A/? That makes package management a lot easier-- might be able to do package management without a special purpose utility. But doing it that way, just one problem is the PATH environment variable has to be huge, and that will slow the computer down-- imagine working around that by reordering the directories in PATH on the fly-- yechh. Heck, X (XFree86) does it both ways now: most stuff under /usr/X11/, except for the stuff now in /etc/X11/. Why didn't they use /usr/X11/etc? Anyway, giving files "attributes" such as {"usr", "A", "bin"} in an unordered set at once solves the problem of which order to use by getting rid of the requirement to have an order. Also, a flag (rwx and others) would be just another attribute.

We have "namespace pollution" problems, and informal conventions to deal with them. (An example is the bzip2 library, which originally used names like "compress" and "decompress" but changed all the names by prepending "bz2".) How do programmers learn of these customs? Burned hand method, that's how.

Which goes nicely into the 2nd major complaint: C.

Nice to have one language to do it all, except C can't. (I can already hear it: use LISP!) C comes close, and it was brilliant to do all the UNIX software in one language. I think a big reason C beat out Pascal/Modula is the syntax is shorter-- "{" instead of "begin", for instance. But C isn't so brief for other tasks, so we have the makefile "language" and shell scripting (csh and tcsh tried to be like C), to name 2 of the ugly bandaid solutions. This duality leads to the practice of having a "frontend" to use in scripts and from the command line, and a "backend" of library functions to call from C. Why have 2 separate interfaces?

C has another kind of problem: the buffer overrun and similar ilk that arises from the C philosophy of "trust the programmer" and don't make checks because it slows things down. How about the infamous gets() library function? Makes for buggy and insecure software.

C and trees: they were great, they've worked for 30 years. But they're not perfect, and now, after such a long time, we've seen a lot of shortcomings and thought of a lot of improvements.

Yes, I agree that filesystem abstraction is good. Have you looked at the Reiser4 filesystem? The papers on that site have similar ideas to yours. Why can't the filesystem be the interface to many things? Then we gain in the ability to use common tools (ls, less, rm, ...) on something new.

Also, I am aware of two projects to create filesystems in Linux userspace. This is also a great idea and I don't truly understand those who do not see the usefulness. One is FUSE, the other is LUFS. Both require a kernel module (obviously), but from that point on users (other than root) can mount and use arbitrary filesystems. I have used sshfs included with LUFS to mount a remote fs only available via ssh. This makes me so happy. I love running commands over ssh, but I love running local tools on remote files even more.

>there are efforts to finalize Reiser4

According to the ReiserFS 4 page, it's released.

What do you mean when you say there are efforts to finalize it? Are you talking about bug fixes? Even if you are, I don't think you should lump it in with WinFS since ReiserFS 4 is available now, and WinFS ain't (and won't be for several YEARS).

I can. Did it 1 or 2 years ago with PostgreSQL 7.something on a Linux box, on a Reiser 3.6 filesystem with patches to enable full journalling on the database filesystem. I treated the machine like an old fashioned DOS box and simply turned it off when I felt like it. Short of hardware failure, I could not kill it. In the boot that followed it committed all the disk activity it could and the service simply started with the last (partial) transaction cleanly rolled back.

At the same time, I could not kill the Oracle database I installed on the same filesystem (though it only took 1 interruption to render it corrupt on ext3 with only metadata journalling). The only difference then is what happens when your admin screws up and blows away important files and you need to restore from backups.

Of course, Oracle has had that under control for quite some time - you can roll back/restore data to whatever point in time you like if you have taken sufficient backups, redo and archive logs. Now PostgreSQL has just introduced something to provide similar ability (I want to see revised manuals!), and appears to be a real option. I'm itching to play with PostgreSQL 8 on a Reiser 4 filesystem if I get the chance! (I believe still on Reiser 4's TODO list is an API to turn an arbitrary set of disk operations into a completely atomic operation. This IS funky!)

It's hard to go past how well Oracle RAC clusters can tolerate hardware catastrophes, but I still have a gripe with it, not so much in the backend as all the Java apps that come with it - yes it appears they were written to run on anything, but the memory sacrifice is far from insignificant. And as for support, well... just make sure the Oracle app is certified to run with your chosen backend!

PostgreSQL on a reliable filesystem with suitable mirroring/striping is simply a rock solid alternative to the commercial platform. It works - reliable and fast. MySQL is yet to particularly impress me (it always seems to be lagging behind the leaders), though it is continually improving.

Who can speak for the others? Open sourcing might be the right thing to do, but the competition is fierce. Hopefully they will remain viable to keep the game more interesting.

Thats how reiserfs works

They should have licensed reiserfs. It uses a block system but small files can share a block:

http://www.namesys.com/v4/v4.html#sharing_blocks.

You can get a special license to include it in your own proprietary OS.

ReiserFS can have up to 8Gb on 32bit systems, and even bigger on 64bit systems. The theoretical limit is 2^60 bytes.
See the reiser faq.

When I joined the company I'm currently with, I found it hard to believe they were using MySQL instead of it, but it looks like MySQL got the foothold first.

Stick PostgreSQL on a Reiser 4 filesystem with full journalling enabled, and barring hardware problems your database server should be as solid as a rock.

Nothing against Ingres here, since it worked for me at Uni in 1994, but it's hard to go past the best of the open source when you're already familiar with it.

Well, for $25 you can ask them.

What do you think of the work Hans Reiser is doing with file systems? How does it differ from and/or improve upon Plan 9? What do you think of his theory that (nearly) all database functions should be done by the file system? What do you think about being able to treat files as directories in order to get to special (or not special) info? Is it useful to be able to treat a tarball as a file when you want to and as a directory when you want to? How about file metadata? Data forks? Do you think Linux, Windows, or Mac OS X will come up with the better database/search-enhanced file system?

You want to change resolutions or monitor profile? Reboot.

No, restart X. Or run a second X server. Or hit Control-Alt-Numpad+ to go to the next resolution setting. Or use an X server that implements the RandR extension. (I think most drivers do in Xfree86 4.3).

You want to change font DPI? Reboot.

No, restart X. Or run a second X server. I admit that I don't know of an extension that can change the font DPI on the fly. You may be able to do some magic by restarting the font server to get different fonts in all new applications, but I haven't tried it.

Your system has scheduled fsck to run, either out of routine or out of improper shutdown? You need to reboot after it's done.

One word: ext3. Or another word: XFS. A third word (although some people consider it an obscenity, myself included): ReiserFS. There's even the IBM option.I've been running ext3 on my laptop over three years and many unclean shutdowns, and I never have to wait for an fsck, or reboot afterwards.

About the only thing that requires a reboot these days is a kernel or hardware upgrade. You go ahead and keep on rebooting the system, if that's all you know how to do. Just don't confuse your lack of troubleshooting skills with a deficiency on Linux's part.

We all see what we want to see, I suppose. How about:

- Konqueror's KIO abstracted protocol interface
- Extensively reliable plug-in based software (IM, firefox, etc). Do you remember what generic software extensions were like in 1995?
- Dancing tree filesystems
- MPEG4/divx/ogg vorbis, theora
- Hashing-based multipart/swarming P2P clients
- Freenet - New Linux VMs and schedulers
- Mouse gestures
- Bayesian spam filters
- MOSIX

No innovation? Open your eyes.

Maybe until recently I would have agreed. However, recently I have started noticing projects relating to desktop search: Dashboard, Beagle, DBFS, etc. This is the hot new area of desktop innovation, and at the moment it seems Gnome has an early lead, with KDE looking to start up some related projects too. Off the desktop, Namesys is moving towards related technologies on the filesystem level. To the extent that MacOS-X can be said to be OSS, they have their Spotlight technology. On the other side, well, it looks like we won't be seeing WinFS in Longhorn afterall.....

Maybe until recently I would have agreed. However, recently I have started noticing projects relating to desktop search: Dashboard, Beagle, DBFS, etc. This is the hot new area of desktop innovation, and at the moment it seems Gnome has an early lead, with KDE looking to start up some related projects too. Off the desktop, Namesys is moving towards related technologies on the filesystem level. To the extent that MacOS-X can be said to be OSS, they have their Spotlight technology. On the other side, well, it looks like we won't be seeing WinFS in Longhorn afterall.....

<a href="http://www.namesys.com/benchmarks.html">Reis erfs</a>

<URL:http://www.namesys.com/benchmarks.html>

<a href="http://www.namesys.com/benchmarks.html">Reis erfs</a>

<URL:http://www.namesys.com/benchmarks.html>

This sounds a lot like Reiser4. Metadata, transaction support, lots of other stuff. All there. link. Hopefully, this will make it into the kernel someday, although it might be 2.7 material.

In any case, Reiser6 is where the real database stuff will come in... Reiser4 "only" allows for arbitrary, fast, metadata under each file. As well as having great performance if you don't mind the CPU usage and don't need to delete lots of small files. Reiser6 - more or less the same thing as winfs; beats Storage in lots of ways, the main one being not using an SQL backend, although you might layer one over it.

Just to use your example for one type of problem that NTFS/FAT16/FAT32 users have just now (although there are several types of problems if you think about it for a while).

You have some mp3s for a band called "Green Day" Do they go under emo, punk, rock (or even 'pop'). You may have strong feelings one way or the other as to which category they fall under, and therefore be able to save these files in one place and find them again at a later time. But will other people who use your computer/network have the same feelings about what kind of music green day play? How will other users now find those Green Day mp3s if they dont know which directory to look under?

This is at the heart of the arguments behind metadata and multiple inheritance, The reiserFS home page has lots of good information on the issues involved with file systems

I disagree with the statement that "Every filesystem is a database at heart."

Most filesystems store files in block aligned files, meaning that they are poorly suited for storing phone number sized objects. The phone number sized objects that they do store (owner, permissions, etc.), are usually statically allocated with one off data structures, and very rigid in their implementation details.

Reiser4 tries to live up to your description, but we are the oddballs in the business.

why don't the nice KDE people and the nice Gnome people work on developing a library that sits on top of this and then we can stop all the stupid name calling and use the right tool for the right job

The current packaging systems are not granular enough. A package should contain exactly 2 files, 1 for meta and the other for content - content being text, image, library, whatever. The system should be more like a relational database where it is properly normalised. That way the same data is not replicated all over the place like it is done now with the GPL licence on my Debian system and I would have the ability to update 1 file at a time which would be huge bandwith saver.

Another nice feature to the package manager would be a Reiser4 plugin. This plugin would take the package's meta information and store it as meta files on the filesystem.

r.

Isn't all information potential file data? Is Microsoft really doing something different than has been done before?

The article also states "WinFS uses a direct acyclic graph of items (DAG)."

The math goes back to the 1970's, as referenced by MathWorld Old math can be used in new ways. Is his a new way when it's used in the FS that Microsoft is attempting?

The articles also says: " the WinFS data model provides the following concepts to describe data structures and organizations: * Types and subtypes. * Properties and fields. * Relationships. * Constraints. * Extensibility. "

Does the new Reiser4 file system support any of these concepts? -- Is WinFS really as new and exciting as the marketing and media says it is?

Thanks.

I hate stupid jokes but here is one:

Now that Reiser 4 is out, Bill can write a plug-in for it and call it WinFS!

Probably most of you whine about how we don't do enough to help the poor, and then here are some hardworking guys in foreign countries struggling to pull themselves out of their poverty, and you want to close the door on them.

Go read Adam Smith's "The Wealth of Nations , and you will discover how trade barriers simply impoverish both sides of the barrier.

Stop thinking of just your own greedy selves, and think of the welfare of mankind. Think about how it shouldn't have to matter what country a man is born in, he should have the same chance in life regardless.

Shame on you all.

Oh, and yes, I am biased, because hiring Russians allowed me to start my own company without any venture capital (Namesys), and I am a perfect example on a small scale of how globalization is making the US into a corporate headquarters location for the globe.

And yes, I am sitting around in the US doing the menial labor of running tests on the code my guys write for my US customer at its site because I could not get visas for my guys to come here, when I could be designing the next product instead.

I don't see how Americans becoming specialized in being the entrepeneurs of the world is such a bad thing.

This is of course the kind of knee jerk reaction that any programmer would think of when confronted with the problem. You can already see it implemented in windows. Of course, every user's first reaction is to disable the searching because no one cares about fast searches of their computer, and everyone cares about their system resources. So this is a terrible tradeoff.

OTOH I think Hans Reiser has it right, just look at his vision. Built search from the filesystem up, and it will revolutionize how we think of data.

There is a nice whitepaper on the future of reiserfs and other filesystems for Mac* and Win* OS. The question is: at what level do you need to implement efficient search capabilities ? Filesystem ? Userspace ? Both ?

• Reiser4 is the fastest file system, and the benchmarks prove it.
• Reiser4 is an atomic file system, which means that your file system operations either entirely occur, or they entirely don't, and they don't corrupt due to half occurring. We do this without significant performance losses, because we invented algorithms to do it without copying the data twice.
• Reiser4 uses dancing trees, which obsolete the balanced tree algorithms used in databases (see farther down). This makes Reiser4 more space efficient than other file systems because we squish small files together rather than wasting space due to block alignment like they do. It also means that Reiser4 scales better than any other file system. Do you want a million files in a directory, and want to create them fast? No problem.
• Reiser4 is based on plug-ins, which means that it will attract many outside contributors, and you'll be able to upgrade to their innovations without reformatting your disk. If you like to code, you'll really like plugins....
• Reiser4 is architected for military grade security. You'll find it is easy to audit the code, and that assertions guard the entrance to every function.

Reiser4, for when only the best will do!

---

In my ignorance I thought Unix fs should remain more or less the same.

Then I read the introductory page on reiser4 at http://www.namesys.com/v4/v4.html, and look, here are people that know how to make Unix more Unix.

I think, that is the right direction, and the elegance and semplicity of the ideas described makes it self evident.