File System Round-Up Interview

Little Sheep writes: "An interesting round-up interview regarding modern Linux filesystems is published by OSNews, featuring the developers behind IBM's JFS, ReiserFS and SGI's XFS filesystems."

Comments

Hmmm...

[mirko]

Not much about Ext3, even though Redhat seems to prefer it to the others...

Re:Hmmm...

[windi]

Just what I've been thinking. It should have at least been mentioned in the /. article. Somehow I think that due to the fact of Ext3 beeing completely community build and not having a company representing it, it was kept out of the round-up, even though, IMHO, Ext3 is more mature than JFS abd XFS. Only ReiserFS beats it.

But still, it's going to be interesting as the fs's develop and make Linux even better for enterprise level computing.

Re:Hmmm...

[Anonymous Coward]

how the hell did the above become offtopic ???
you've got to prove me it's offtopic, mister anonymous moderator!

Re:Hmmm...

[Billly Gates]

Perhaps this might explain it. I don't know about you but I think I will keep reiser away from my mission critical servers for now, regardless if this is just a flame war.

netbench recent results

[johnjones]

there are results from
http://lse.sourceforge.net/benchmarks/netbench/res ults/august_2001/filesystems/raid1e/README

I quote

Hello all,

I recently starting doing some fs performance comparisons with Netbench
and the journal filesystems available in 2.4: Reiserfs, JFS, XFS, and
Ext3. I thought some of you may be interested in the results. Below
is the README from the http://lse.sourceforge.net. There is a kernprof
for each test, and I am working on the lockmeter stuff right now. Let
me
know if you have any comments.

Andrew Theurer
IBM LTC


seems that its all pretty much rocking and they turn out the same ish even tho they do things differant except riser which sucks and alaways has in my eyes (each to their own)

regards

john jones

Re:netbench recent results

[Anonymous Coward]

You're a Deltic? Wow, I didn't know there were any large flat bodied, bull nosed deisel electric locomotives from the UK posting to Slashdot.

Didn't most of you guys get cut up back in the early 80's anyway?

Re:netbench recent results

[jsse]

Amazingly, in the article Hans Reiser quoted another benchmark tests which show Reiserfs is superior.

The results too perfect to believe.......may be I'm thinking too much....

Comments...

[Nissyen]

I knew the comments were getting more useless as time went on, but the majority of the first 9 posts are not only off topic, but offensive as well.

Re:Comments...

[l0wland]

Just put your Forum-treshold to +1 and you won't notice any of them.

Re:Comments...

[Nissyen]

Yeah, my threshold seems to go up by 1 every year. These days it's hard to stick with the moderating guideline to try to moderate comments up. I spend all my mod points moderating down off-topic posts.

My experience

[famazza]

I tried once ReiserFS with Mandrake 7.1. That was great and I really felt faster. :o)

But a month later when I tried to recompile my kernel I had a terrible surprise. As ReiserFS wasn't officially supported by the 2.2.x kernels I couldn't make it boot. So I had to draw back. (a month later I installed Slackware 7.1)

Well, I was a newbie then. But this bad expierence with new filesystems make me think twice before install something other than ext2, or any not-officially supported filesystem.

Re:My experience

[Dimensio]

My NAT router is running 2.2.19. When I was setting it up I did a quick search and found several links to a ReiserFS kernel patch, which I was able to apply and compile without a hitch.

And ReiserFS is officially supported now, at least for the 2.4.x kernels. It's also nice not having so see fsck take so long in the unlikely event that your machine goes down improperly.

I've only had one problem with ReiserFS. I mount my /home directory on a different physical disk than my root filesystem, and for awhile after formatting and setting it up I would get inexplicably delays when I attempted to remotely access anything on it;connecting via ssh or ftp would make it access my user home directory and it would pause for several minutes and I have a SMB share on there as well, which would often time out when an initial connection was made.

I backed up the contents of /home, reformatted again and everything's been fine since.

Re:My experience

[famazza]

Oh, I forgot to mention. I turned Slackware a year ago. :o)

Have to give credit to Hans

[wowbagger]

Of the three interviewees, Hans knew more about the other guys, he was better able answer what was better about the others then his, and how his was better than the others. The others came off as suits, he came off as an engineer.

Re:Have to give credit to Hans

[Anonymous Coward]

Reiser is definitely a real character and loves his work, actually passionate about his work is a better way to describe his attitude. Its very unfortunate that a good part of the linux community has shunned him and ridiculed his methods.

The ReiserFS as we know it today will be a totally different beast in 2 3/4 when kernel 2.6 is nearing release, and people will love him for it, completely forgetting all the insults they hurled at him just today.

Hans Reiser , my hat is off to you, Cheers and good luck.

Re:Have to give credit to Hans

[Tet]

The others came off as suits, he came off as an engineer.

From following the posts on the Linux Kernel mailing list, I've come away with exactly the opposite opinion. Reiser is the suit, and the others are the engineers...

Re:Have to give credit to Hans

[Anonymous Coward]

Hans Reiser trolling on the kernel list was a necessary evil. Why? Because some of the core developers needed a session in self realization. Hans Reiser was correct from the start, but the core developers had nothing to do but complain. But Hans it he man that made it a reality, no thanks at all to the kernel hackers.

The short spiff of activity that caused so much noise on the list should not be taken out of context, exactly what you're doing now.

BeOS' BFS uber alles!

[deusx]

I really wish someone would include BFS-style attributes in an Open Source file system. Hell, I really really wish my Mac OS X installation had it. Steve Best kind of dismisses the Live Queries it as "similar to a change notification mechanism," since he admitted he wan't really familiar with it... but it's more than that.

In BFS, (although I'm probably going to butcher this explanation) the system actually retains the optimized parsed tree of the query, and monitors the modification times of the individual indices used in the search. When one of those times changes, the system re-queries just that branch of the tree rather than re-processing the entire query. Is very neat. Oh, yeah, and there's some notification going on.

Bah... I just really want a native file system with arbitrary indexed attributes so I can run SQL/LDAP-like queries against my non-Be machines too...

Re:BeOS' BFS uber alles!

[Anonymous Coward]

So port AFS from AtheOS to Linux. Not only would it give you most of the funky features you list, but it would be useful to a lot of AtheOS developers who can make use of it as well.

Re:BeOS' BFS uber alles!

[Adnans]

I really wish someone would include BFS-style attributes in an Open Source file system

It's called XFS :-)

XFS supports extended attributes just like BeFS. no wonder, since BeFS is based, at least in part, on ideas originated in XFS. You will need some userland tools to set extended attributes and probably a modified tar if you want to preserve attributes inside archives. The web page on Access Control Lists and Extended Attributes might also prove useful. Now, combined with fam & imon , which provide (i)node monitoring you have everything except indexed searching, all in Open Source (imon is actually a kernel patch that adds node monitoring at the VFS layer)

-adnans

Re:BeOS' BFS uber alles!

[deusx]

Hmm.. that might be getting close, though the indices with maintenance and searching are pretty integral to BFS, along with the whole query and live query system.

Though... maybe it wouldn't be too hard to work that into XFS. Most of what I know comes from undergrad Comp Sci classes and a few readings of the BFS file systems book.

Re:BeOS' BFS uber alles!

[fm6]

I just really want a native file system with arbitrary indexed attributes so I can run SQL/LDAP-like queries against my non-Be machines too.

I'm all for extended attributes. It makes a lot of sense to put the file system in charge of metadata. Grossly superior to using filename extensions.

Problem is, extensions are what people know, and what almost all Windows, Linux, and Unix apps are built around. Getting everybody to change won't be as hard as retiring the QWERTY keyboard, but it's up there.

And feature creep makes matters worse. Why does a filesystem need to support arbitrary relational queries? Why make every single app deal with complex access issues that only a few will ever use?

Re:BeOS' BFS uber alles!

[ReelOddeeo]

Why does a filesystem need to support arbitrary relational queries? Why make every single app deal with complex access issues that only a few will ever use?

Why would every single app be affected? If you don't want to use a feature don't.

On my Mac, I can search large file systems across multiple hard disks for a file with a certian name or attribute, very fast. For instance, find me all files that end in ".jpeg", are bigger than 3MB, and are in a folder that is locked. I can get results quick.

Much better than using locate/updatedb or find.

This seems like an advance to me.

BeFS and new file systems

[jlmcgraw]

I find it interesting that BeFS is mentioned so prominently by each of the developers as a goal for an FS to aspire to, yet the OS itself has basically died even though it was given away for free. What does this tell us?

Re:BeFS and new file systems

[Anonymous Coward]

So your saying the FAT32 is the best FS as it the most commonly used and you even pay for the OS's that use it naitvely.

Do you even know what BFS can do?

How you ever even used it?

Re:BeFS and new file systems

[Sloppy]

It tells us that it's a real neato piece of work that everyone wants, but that few could actually use, because it only ran under one OS (and it happened to be an OS that saw very little use as a server).

Filesystems, just like data formats and communications protocols, must be open in order to be generally useful.

(Novell's filesystem was able to sort of skirt this problem for a long time, because even though it was closed, it was implemented on a server, so that other OSes could "use" it without actually knowing anything about it. But they did their best to kill it anyway, by keeping NCP closed so that only a few platforms could use the server. If it weren't for sniffers, Netware would have become obsolete a lot sooner.)

Re:BeFS and new file systems

[snarfer]

I find it interesting that BeFS is mentioned so prominently by each of the developers as a goal for an FS to aspire to, yet the OS itself has basically died even though it was given away for free. What does this tell us?

That the fact that you can not purchase a computer that has Windows plus another OS installed on it is the result of illegal contracts foisted on the computing industry by an illegal, out-of-control monopoly?

That really, really, really bad management can kill even the best technologies?

That a smart software publisher should call Palm and see about licensing the BeOS desktop to sell to the public?

Re:BeFS and new file systems

[SpryGuy]

Filesystems, just like data formats and communications protocols, must be open in order to be generally useful.

I dunno. I find NTFS generally useful, and have for years and years...

Re:BeFS and new file systems

[be-fan]

Except not really. BFS is very well documented, and some of its code actually appears in Giampalo's fskit. Plus, all of the data structures are cleanly laid out, so implementing a clone (eg. AFS) is very straightforward.

Re:BeFS and new file systems

[Will Dyson]

It tells us that the article's author, Eugenia Loli-Queru, is a longtime BeOS community member (and married to a [former?] Be employee). So it is only natural for her to think that BeFS rocks.

In fact, the BeFS's convient arbitrary meta-data attributes and indexing are a big reason that I still spend about half my time booted into BeOS. I'm really glad that ReiserFS is moving in that direction.

Re:BeFS and new file systems

[megaduck]

I find it interesting that BeFS is mentioned so prominently by each of the developers as a goal for an FS to aspire to, yet the OS itself has basically died even though it was given away for free. What does this tell us?

This tells us that no matter how technically brilliant your OS is, you need developers. We went through this before with OS/2. OS/2 was much better than Windows in all sorts of ways. IBM failed to get third-party developers on board. No developers = no apps = no users. When Windows 95 came out, it sucked but was universally adopted because Microsoft was able to leverage their existing Windows developers.

Personally, I really wanted to see the BeOS succeed. It was fast, pretty, versatile, and thoroughly modern. I couldn't make the switch, though, because the apps and drivers I wanted simply didn't exist. From what I understand, this was because Be didn't work on cultivating those third-party developers that are so important.

For an interesting contrast, look at Linux. Linux is so darn developer friendly that we've got 3+ full GUI environments, 4+ journaled file systems, a bunch of web browsers, a couple of Office suites, and device support up the wazoo. We've even got some decent games (thanks, Loki!). Be didn't have any of that, despite their incredible technology. Again, developers = apps = users.

I'm really sorry to see the BeOS gone. It really was ahead of its' time. Fortunately, the whirlwind nature of Open Source development means that a lot of the good ideas in the BeOS will probably make it into another OS eventually. Good ideas never die, they just go into hibernation every now and then.

blah

[Anonymous Coward]

Back in my day, real hackers use Minix FS and DIDN'T COMPLAIN. So there.

Re:blah

[spauldo]

Yeah they did. They complained that andy wouldn't add any of the features is direly needed. Why do you think so many people followed linus?

collaboration

[A moron]

Hans Reiser: I'd like to thank the XFS developers for taking the time to personally explain to me why delayed allocation is the way to go.

Now that's cool. Projects that appear to be in direct competition, but they all have great respect for each other and actually communicate with each other. And in the end, each product ends up being better.

Corporations take note.

Re:collaboration

[mikeage]

I admit I don't know if that's true or not... but I just can't shake the thought that this is not really true...
(-1 Insulting)
(+2 But it's to a big company)
(-3 They have an awesome employee server)
(+4 But they shut it down (reality.sgi.com)

Re:collaboration

[linuxpng]

perhaps you haven't notice the pure hatred Hans has for redhat. Ever opportunity he gets he criticizes redhat's choice for GCC (which is undefendable as far as 7.0 is concerned) and seems to be at war with Alan Cox. I'd hardly call it friendly.

Production Use

[Marvin_Runyon]

I think the clearest choices for a production environment are XFS and JFS, as they have many years of proven reliability.

Generally, datacenter environments demand reliability over speed, and a good track record wins the day in selecting technologies.

When we implemented our database backend, we decided to go with SGI's JFS based on years of production use. So far we havn't encountered any problems!

-Marvin

Re:Production Use

[Anonymous Coward]

Well they dont have 'many years' of proven reliability on linux ... many parts of each fs had to be rewritten because they were either too centric to their host operating system... i wouldn't trust them too much, just quite yet

Re:Production Use

[Raptor-X]

Umm you mean IBM's JFS - SGI made XFS :)

Re:Production Use

[Anonymous Coward]

i wouldn't trust them too much, just quite yet

Got anything to back that up, or are you just letting us all know about your "hunch?"

Re:Production Use

[FlyingDragon]

Got anything to back that up, or are you just letting us all know about your "hunch?"

When it comes to handling important data, he doesn't need to prove they're untrustworthy -- they need to prove they are trustworthy. The ball is in their court. By the same logic, I don't trust wu-ftpd or sendmail as secure. I may not know of any current exploits, but they have a long history to live down.

Pity ext3 is the de facto standard

[thejake316]

For what it's worth, what RedHat says generally goes. Also, I can live with my OS being named after somebody, but I'll be damned if I'll sit back and let my filesystem be named after that guy on Mad About You. What's next, HelenHuntinetd?

Re:Pity ext3 is the de facto standard

[ReelOddeeo]

For what it's worth, what RedHat says generally goes.

Really?

I tried RH 5.2 as my first Linux. Hated it. No KDE. Other complaints.

A friend pointed me to SuSE. Boy am I glad. I might never have given Linux a second try.

I've been using SuSE for years now, completely ignoring what RH says. I'll confess I've been using KDE for several years. And ReiserFS for awhile too. But don't tell RH. Or MS.

It's nice to have choice.

correct me if I am wrong,

[Anonymous Coward]

but I didn't even think Reiserfs even journaled.

Sad technical loss

[sphealey]

One of the highest-performance, most managable, most securable file systems on the market for the last 15 years has been Novell's NCP file system for Netware (I am sure it has a name but can't think of it at the moment!). The current versions (Netware 4 and 5) are supurb techncial achievements.

Now, it seems pretty clear that Novell is doomed, and when it goes Netware and NDS will evaporate. I just hope that whoever turns out the lights in Provo has the foresight and generosity to release the details of those two technologies under some sort of open source license, so that even if the products disappear the technology might be saved.

But I doubt that will happen.

sPh

JFS in the kernel?!?!?

[Ignacio]

Yeah, right! Have you seen the code for it? It is a horrid mess. I tried digging through it, and soon found that I would have to rewrite it from the ground up. I'm not surprised LT et al. don't want it in Linux. It's a nice FS, but it's NOT ready to go into the kernel.

Re:JFS in the kernel?!?!?

[Anonymous Coward]

been there, done that. you're absolutely right. the code for JFS is a mess.

Hmm...

[slaytanic killer]

riser which sucks and alaways has in my eyes (each to their own)

Those numbers sound very suspicious. Even though the "default" config was used, I imagine that possibly the tester used a distro like RedHat which turns on debug mode, slowing down Reiser FS. When testing databases, you see shrinkwrap agreements which forbid you from publicly posting benchmarks for this reason.

He mentions he used:
linux 2.4.7, Samba 2.2.0, and NetBench 7.0.1
and it is unclear whether that was from an upgraded distro or not.

ReiserFS stability

[efgbr]

I think it's very strange to see Reiser saying SuSE is stable and Red Hat is not when we all remember the problems people had with it when running SuSE.

ease of and quickness of use

[nilstar]

I think that my choice would be ReiserFS (which I've been using for about a year), because of its inclusion in the kernel.... and the way that the nice people at Mandrake Soft. have packaged it. I mean from the get-go, a fresh installation, you don't even need to have more than a 20mb /boot partition with ext2, the rest can be reiserFS.

Also, migration to ReiserFS is pretty easy... check out this link.

Re:ease of and quickness of use

[Dimensio]

Why can't you have everything, including / and /boot, be ReiserFS? True, 20MB isn't too bad for fsck and even if the partition gets trashed it's pretty easy to store a backup somewhere, but I do like consistency and I like keeping as few partitions on a physical drive as possible.

When SuSE first started supporting ReiserFS I installed it and it let me put everything on a Reiser partition...too bad the install only supported Reiser as a module :/

Re:ease of and quickness of use

[asincero]

> Also, migration to ReiserFS is pretty easy...

Couldn't be easier than ext3 tho. All that involves is specifying the -j switch to tune2fs.

- Arcadio

Re:ease of and quickness of use

[nmarshall]

you can have /boot on ReiserFS, but if you use GRUB and make a small /boot partition you will never have to mount /boot other then to add a new kernel.

This way you can always boot up. even if / gets fubared

Re:ease of and quickness of use

[Billly Gates]

My choice would deffinetly not be reiser. Its too cuting edge and imature for server use. This particular bug may have been fixed but in a server environment, reputation for stability needs to be there. Its not like I can bring the server down and download a patch to fix a fs bug in a mission critical environment. XFS has been around for years and would be my choice if I had to bet my job on a fs.

Re:ease of and quickness of use

[Cadrach]

As a matter of fact, you can boot from ReiserFS. I have nothing but ReiserFS (and, unfortunately, FAT 32) on four computers running various 2.4 kernel versions. The computers were originally running Mandrake 7.1, 7.2, or 8.0, but they have since been modified to varying degrees. ReiserFS has worked great for me, but I have no experience with the other journaling filesystems.

Re:ease of and quickness of use

[omega9]

Speaking from memory here, but I believe the partition has to be certain size to use ReiserFS because of the journal. I.E. - you couldn't have a 8meg ReiserFS partition.

This usually comes up with /boot partitions because they're usually small.

Dyslexic

[Anonymous Coward]

I suppose he's Dyslexic, which is very hard for people of that nature to spell.

Most Dyslexics that I have worked with have had better maths skills and are more logical even if their grammar and spelling leave a little to be desired.

the original AC

JFS pulled from Mandrake 8.1

[LinuxGeek8]

Interesting review.

On MandrakeForum the latest news about filesystems is that JFS will be pulled from Mandrake 8.1.

There was done a test with a buildup/takedown of 100.000 files.
In the case of JFS the deleting of those files caused a hard kernel crash.

Seems there is some work to be done, despite it being a 1.0 release.

And hey, what's up with this html here?
Seems only plain text works right for me.

What about tux2?

[DJK]

I remember reading on /. about another filesystem called tux2 .

The web site is here, but I don't really see anything giving a status update.

Anyone know anything about this project?

reiserfs on 2.4 vs 2.2

[TheGratefulNet]

I've been running the latest reiser on the 2.4 series, both at home, at work and on my laptop (especially laptop, since the battery can go out at any time with little notice. fscking a 32gig notebook (slow) drive sucks...)

I had some problems I thought were 2.4 related so I tried the 2.2.19 kernel with reiser. it didn't seem to be able to locate superblocks when it booted up and just froze ;-( so I had to go back to 2.4

this is the only downside I see. I know 2.2/2.4 reiser should work but it didn't for me. and I think I did all the right things when building the kernel AND userland tools.

I've not tried xfs and I did have some hangs with jfs when it reached 1.0.0 - but that was while on an IDE system with the infamous VIA chipset. I'd probably blame the chipset bugs before JFS, but it did shy me away from JFS.

reiser seems stable on 2.4 and I'm quite happy with it. give it a try.

Re:reiserfs on 2.4 vs 2.2

[Anonymous Coward]

I have to second some support for ReiserFS. If I leave the house without turning off my computer, my mother has a nasty habit of hitting the power-switch. My average uptime is only about 8hr, and I've never had a crash.

If it can survive my mom, it can survive anything.

Re:reiserfs on 2.4 vs 2.2

[rtscts]

The 2.2 version of RFS is compatible with 2.4, but the 2.4 version is not compatible with 2.2. You have to re-mkfs your filesystem(s) if you want them to work with 2.2 - you'll have to read the docs yourself to find out what the mkfs parameter is for 2.2 compatibility...

Re:reiserfs on 2.4 vs 2.2

[TheGratefulNet]

thanks for the info - I just assumed it was backwards compat. my goof.

now, if there was a convert util that did the re-mkfs in-place, I'd be ok with that. but I'm not about to reconvert (by hand) my entire set of reiser partitions. oh well - guess I'm stuck with 2.4 then.

Re:reiserfs on 2.4 vs 2.2

[spuk]

Reiserfs on 2.4 uses format 3.6, while (I think) 2.2 uses 3.5. But, from what I understood of the docs, using the 2.4 reiserfs on an old 2.2 filesystem does no harm, and maintains it backward compatible. You can use an option to mount it converting gradually to 3.6, in which case it will no more be backward compatible with linux 2.2 reiserfs.

I've been using a reiserfs partition interchangeably between kernels 2.2 and 2.4 with no problem at all, without the 3.5 -> 3.6 conversion.

Don't forget tux3

[AeiwiMaster]

info
http://people.nl.linux.org/~phillips/tux2/

ext2 patch
http://people.nl.linux.org/~phillips/htree/

Bastard non-journalling filesystems.....

[Anonymous Coward]

I'd recommend any of these, if you thought not having fsck was the only advantage check this bitch out.

In short my dialup router was using ext2, it got hard rebooted in a storm, and changed my dialup access number it was dialling... later when the bill for £1000 arrived I was enlightened as to why I should have put reiserfs on that box aswell as my workstation....#%$@!


There are no stupid questions, just lots of inquisitive idiots. taflap.

Re:Bastard non-journalling filesystems.....

[Anonymous Coward]

Hah. Good troll. No way a storm would change the phone numbers in you dialing list.

the reason is small files

[johnjones]

hey no those results are right you just have to put them into context

Reiser is desgned for large amounts of Dir and files which is what this tests for

and yes it achive very high marks but I have yet to see a TB on a Reiser in a live platform yet I see one every day for XFS (streaming video) and I know someone who has AIX with 3TB on it

most large files seem to be of the video nature or are database yes while MP3s are comman and thata is what Reiser did (remember mp3.com was a sponsor and came up in the boot up) but I have to say round here most peoples dir do not contain over 1000 files most have some MP3s and video, documents and the like

really I have been running XFS on a i686 for a while and have not had anything go wrong (we havent exactly pushed it tho)

what XFS and JFS need are ports to other archs and JFS seems to recogise this

remember benchmarks are written to test certain things but we commanly relie on quantum chaotic things and are unable to test for this (because they dont really have random things)

Metadata APIs?

[Sloppy]

One thing that pleases me about this, is that it looks like all three of these filesystems have (or will have) metadata support. This has been a pretty serious (imho) weakness in traditional Unixes. (BTW, isn't it interesting that the platforms with the best GUIs (MacOS and OS/2 WPS) happen to depend heavily on metadata?)

I haven't used or programmed for any of these filesystems so far, though, so I'm wondering if the APIs for getting at the metadata stuff, are all the same. This is something that will be absolutely necessary before Linux app writers will be able to start using this stuff.

It looks like the teams have good attitudes toward one another. I hope they're coordinating on the APIs so it'll all be consistent.

Re:Metadata APIs?

[battjt]

I think that having a filesystem store meta data is stupid. (Now that I have you in the right frame of mind for a civil discussion!)

A filesystem is the ONLY place I have to store information in most OS's (so, I'm discounting raw devices). It should have the best performance for the lowest common denominator. If you want complex meta data, use a 'data'base. Databases are designed for storing and searching highly structured data; filesystems should be for named unstructured data.

If you think that a database is over kill for most applications, then define a database for me. Is Oracle a DB? DBM? XML? .INI file? A lisp file? I would argue that every file with a know structure is a database.

We do need to know what format the data is using, so the only additional peice of information that I need that is not in current metadata is the MIME type.

Joe

Re:Metadata APIs?

[spitzak]

ReiserFS is supposed to suppor this.

However I feel that metadata is a bad idea. The problem is that one of the main things done with a file is to copy it from one place to another. In fact with the web, I would say that non-system files are copied hundreds of times more often then they are "used".

This requires the metadata to be encoded and transmitted with the file. These encodings tend to be complex, kludgey because they are never designed for completeness, and require both ends to understand the encodign. And the end result is a stream of data that most programs (all those concerned with copying files) will treat as the file's data anyway.

There is also the annoying fact that all programming languages need functions added to them to manipulate the metadata.

So how about imbedding all that metadata into the file's data? I have thought that a system could be designed where a file is *only* a block of data. Even the file's name and protection is imbedded in the file. A filesystem would be designed so the first 1K or so of a file is very quickly accessible. There would also be simple rules so that the data could be located and it would appear as comments to whatever reads the file.

Since anybody with write access could change the permissions, permissions would actually be some and/or combination of that file and all the parent directories. This is the only complex part I see to this.

Re:Metadata APIs?

[Skapare]

The very fact you might need a new API would be one major reason why metadata is a bad idea. While the concept of having metadata is not in itself bad, implementations of it that require special access are. For example, if I have a filesystem with metadata that uses special API system calls, how will I be able to backup and restore that filesystem with the metadata using ordinary tools like tar?

A simple way to have metadata is to define another file. Just append ".meta" to a filename, or replace its extension with "meta". You can now read and write as much metdata as you want to have. Then any filesystem backup or transfer tool can do the job.

Re:Metadata APIs?

[ReelOddeeo]

filesystems should be for named unstructured data.

I agree that the data part of a file should not have any structure known to the filesystem. The structure of the content of a file is unstructured.

But I disagree. You said "named unstructured data". The name is metadata. There seems to be no disagreement that the name is useful. (C'mon, why can't we just all use inode numbers and do away with names?)

The metadata is to make files easier to find, and to store more information about what is in that unstructured data.

We do need to know what format the data is using, so the only additional peice of information that I need that is not in current metadata is the MIME type.

You mentioned filetype. I agree. But I don't think we should stop right there. I think "creator" is another good one. (i.e., which application will launch when I click on this file? e.g. multiple "jpeg" files might launch with different applications, and accordingly display different icons.)

Another good piece of metadata about files is their X,Y location within the window that displays them. Think about this. The Macintosh is the only OS I know of that has a geographic memory of where you put a file. I can remember, "oh, yeah, when I open that folder, I left my business plan in the mid-lower-left corner of the window."

There are other useful metadata as well.

This does not mean that existing apps have to be changed at all. On the Mac you can still write a C program that is totally ignorant of the Mac's extended capabilities. You end up with users saying "why is this program so stupid? why do all it's files have plain white icons?" But the program works. Your C program doesn't have to set the X,Y location of files, the system does that, when the user moves the icon for the file.

On Mac, every file and folder can even be given a custom icon that overrides the icon the file would display based on the combination of it's type/creator. This is another good use of metadata. In fact, it begs the question, if the fielsystem can associate metadata with a file, then why not just make the metadata completely arbitrary? The FS doesn't need to know what attributes KDE or GNOME want to tack onto a file. A window manager could tag a file as having a certian "X" attribute, and certan "Y" attribute. I custom icon attribute. etc.

But you could still type: vim mypict.jpeg, and open a jpeg file using a non-aware application.

Just because the filesystem supports extended attributes doesn't mean you have to use them. Nor do you have to run a GUI that takes advantage of them. Nor do you even have to run the particular filesystem that has extended attributes.

Some of us want better GUI's.

Re:Metadata APIs?

[Sloppy]

if I have a filesystem with metadata that uses special API system calls, how will I be able to backup and restore that filesystem with the metadata using ordinary tools like tar?

Well, unfortunately, you can't, unless you modify tar. (This isn't all that far-fetched; I used to use an OS/2 port of tar that saved and restored extended attributes.) Of course, that only works when tar knows it's working with files, and the approach falls on its face when you're just doing streams. So yes, there's a problem.

Just append ".meta" to a filename, or replace its extension with "meta".

This does mostly work, and it's what my Amiga does (.info files), and OS/2 does something a little like this when you try to use extended attributes on filesystems that don't support it (e.g. FAT). The main problem with this approach is that the files can too easily end up getting seperated from their metadata. You still need non-standard (from a Unix point of view) tools that know that when they copy a file, they have to copy the file's metadata too. "cat foo > bar" isn't going to copy foo.meta.

Tux

[johnjones]

this looks to be a great system

I use a nettapp all the time and a filesystem that had the same sort of funtionality would be great

I heard about this in the write up for linuxconf.au but heard little about it since

regards

john jones

http://people.nl.linux.org/~phillips/tux2/

JFS Working Nicely...

[PingXao]

... on about a dozen systems I administer. These boxen are RH7.1+ with 2.4.3 kernels. I haven't taken the ultimate step of moving root partitions onto JFS yet, but for everything else there hasn't been one problem in 6 weeks.

Another poster commented that the source code was rough around the edges and therefore didn't merit kernel status. I disagree. The reason I don't move some of the boxes to using JFS for the root partition is precisely because it's not yet in the kernel. Thus, kernel upgrades become dicey if you can't smoothly apply the patches. Also if something goes wrong you're basically up the creek. Of course, there are backups, but that's an amazingly tedious exercise.

As for the code, well, isn't that part of the beauty of open source? At least you know what you're getting. And I've seen plenty of code - albeit not necessarily in the kernel - that looks like complete garbage but works great.

Re:JFS Working Nicely...

[Ignacio]

The reason JFS isn't in the kernel isn't because it's "rough around the edges", it's because there is no clean separation between the JFS utils, libs, and kernel code. If you look at ext2, you can easily compile its utils even on a Linux machine that doesn't use ext2 (not that I've seen too many, mind you...). Try the same with JFS. Go ahead, I DARE you.

References on filesystem design?

[torpor]

One thing that's always interested me is the technology behind filesystem designs - are these guys operating on any references, that might be worth studying? "Filesystem design 101" sort of books?

Anyone know?

Re:References on filesystem design?

[Anonymous Coward]

theres one i know of...called something like "Be Filesystem Design". i cant remember the name exactly though.

Re:References on filesystem design?

[be-fan]

Practical File System Design by Dominic Giampalo.
Great book, has a detailed overview of the BeOS structure, some comparisons and explanations of other filesystems (like ext2, NTFS, XFS, and HFS+), and a cool file system builder kit with full source to pretty nifty example filesystem.

Re:References on filesystem design?

[Anonymous Coward]

Unfortunatly, there aren't many good books on file system design. I asked the same questions about a year ago and got this answer. In fact, there seems to be only one book on the subject that you are likely to be able to find. Practical File System Design with the Be File System. Although it's written by one of the developers of the Be file system, it uses that concrete system as more of a touchstone.

It's a well written book. If you're interested in the topic I suggest checking it out the next time you're in a bookstore.

All journall FSs useless on Linux

[Anonymous Coward]

Journalling file systems depend on the correct order of data
written to the disk. Simply said the journal must be updated first and then
the actual data. To achiee this , the linux kernel must support ordered
writes. I don't know if it does or not. It probably does, because
it would be really weird if not :-)
But there is another problem. Write cache in the disk drives. Linux
sends the data in correct order, but then the disk firmware
reorders it and if you pull the plug in the middle of write,
your journalled FS has gone to hell ...

And the linux developer don't even care about this. I got responses
like this on LKML :
- turn off write cache if this bothers you ( this
will impact performance and is not even possible
on some disks , they keep using the cache to look
better on benchmarks )

- it is not possible to solve on IDE drives , so there
- we don't feel like dealing with this right now ...

stein
( email address at http://surf.to/stein )

Re:All journall FSs useless on Linux

[Skapare]

So what is your solution?

Re:All journall FSs useless on Linux

[rabidcow]

I don't know that it would fix AC's issue, but personally, I would like to see more done with the phase tree algorithm used in Tux2.
Write order seems to be less of an issue there.

All your journal are belong to us

[Skapare]

So when do we get to see a complete round up of all journaling filesystems (I noticed ext3 was missing), including comparisons of features (including implementation features) and benchmarks on a variety of hardware configurations (SCSI, IDE, USB, RAID)?

Re:All your journal are belong to us

[Eugenia Loli]

Yes, ext3 was missing because ext3 is nothing but ext2 with journal. I wanted my article to be closely to the more modern filesystems.
For the record, my personal best from the 3 of them btw, is XFS.

IDE drives are for kiddies

[Anonymous Coward]

What you say has nothing to do with Linux and everything to do with IDE drives. Every OS has the same problem with IDE drives. Solution? Be a man, and go SCSI.

Oh, and with respect to ordered writes, ReiserFS uses the ATT/Bell/USL/Caldera patented ordered write alogrithm so it is obvious you are blowing smoke out your ass.

Fuck you nigger boy

[Anonymous Coward]

Niggers are the scum of the earth. Niggers like you should be sent packing back to your "roots". Adios, jigaboo.

Ext3 carries Ext2's quota support...

[NRAdude]

From what I know, all the utilities for setting user and group quotas, on a Linux system, are built around the Ext2 filesystem. As a simple, educated guess, that's way RedHat prefers Ext3; because there is less work on their part in implementing quota support. I know for a fact that IBM's ReiserFS doesn't support quota and has no utilities to configure this. Reiser is planning this in the future, but know date has been released on its release. As for SGI's XFS, I have no experience, or idea, of how XFS works. I never got around reading the databooks. I heard a rumor from an AC that it supports quota, but I can only speculate on a rumor which is not worth flapping any more keystrokes over IMHO.

Slashdot didn't mention anything about Ext3 possibly because it is Ext2 with journalling; not much to talk about maybe? It may be less-featured, generic, compared with ReiserFS and XFS.

I doubt it

[spuk]

I don't believe it. XFS is designed totally for performance, and still it pairs with ext3!? Not even ext2!