Slashdot Mirror
Reiserfs Released
Kewlname writes "Namesys released Reiserfs for GNU/Linux. The specs and the press release " To think - we get this and beta journaling code in the same week (I hope!). Anyway, I don't know how applicable it is to Reiserfs, but
Theodore Ts'o has a paper in the 1998 Linux Expo proceedings about how one might go about introducing B-trees into ext2. It might be
interesting to compare this to the design of Reiserfs.
Is Mr. Reiser on some sort of ego-trip so that he named the fs after himself? How does he think other people will contribute free fixes to an fs that selfishly bears the name of someone else? 'Linux' is similar to it's creators' name too, but there is a big difference: Linus did not pick the name himself.
I thought Linux 2.2.x introduced enhanced directory cacheing that does this. Linux 2.0.x would cache directories but would have to be searched through sequentially for every lookup. Is this true or am I mixing things up again?
Sure. It's very simple. Use ext2; it's a proven solution. All the others are brand new (alpha!) code and probably will introduce filesystem corruption. Stick with the proven, stable ext2 for at least the next 12 months.
XFS is NOT an option. If you don't believe me, try compiling a Linux kernel/module that enables you to use XFS on your Linux box.
-T
can we ban anonymous cowards? please?!?!?
Are you sure he needs to ask for it, the modifications _were_ made under his copyrighted code.
In fact, I plan to use it for a file system based object database soon.
I doubt he's trying to sell the Linux version. It's more likely he wants to sell the file system technology involved to vendors of commercial OS's. As for the GPL issues, other people have already dealt with that...
According to the reiser web pages, he changed the
name to reiserfs after an unsuccessful attempt to
get VC in russia - someone was going to bribe some
"scientists" to claim in court that reiser had
nothing to do with it.
Hopefully I have it right... it's in the abstract
somewhere.
Mark
XFS + Reiserfs + b-tree ext2 = New Filesystem???
I seem to remember Linus saying something about ext3 in an interview, but SMP support being more important at the moment. (Or is my memory playing tricks with me...) With all the new filesystems source being released, is anyone working on a completly new one? I think a combination of ideas could lead to interesting results.
news server operators will LOVE this.
The code may be good, but the PR sucks. Even to a frenchman like myself, the language used in the press release sounds clumsy and amateurish.
It's a press release. They all sound like that.
Web servers, newsservers, mailservers (those that doesn't use the mailbox format - an the mailbox format causes performance problems of it own) will all benefit from reiserfs. Probably most fileservers too.
And another point here is that one of the main reasons for using large files in the first place is that filesystems like ext2fs behave in a completely brain dead way when you have many files in a directory, so people tend to design software to use few big files instead. If you've read any of the documentation from the author of ReiserFS, you will see that that is one of the things he doesn't like, and I agree.
I'm actually working on an object database that will store each object as one file, based on the hope that reiserfs would be released before I had something useful, and it did :-)
Hopefully, now that we have at least one filesystem that handles small files gracefully, more applications will appear that doesn't cluster information artificially together to overcome filesystem limitations.
Everything. Look for the benchmarks...
I thought it was up,up,down,down,b,a,select,start
I say this because that code is burned in my memory. While I haven't used it in about 10 years, I can still recall it at any time. This is very annoying. I'd rather be able to swap out facts in that memory space. I know I could happily forget the code.
who cares?
No, this is less of a benchmarks issue, and more of a checkmarks issue.
PHB's love to compare products with matrices that give a checkmark where product A or B has feature X, or Y. A "Journalling Filesystem" is an important checkmark, whether or not there is any real-world benefit (there is- but this is an important checkmark that's been missing from the Linux column on the chart for a long time).
I could beat Contra without dying..
The mac has a b*-tree based filesystem. It has a huge, gaping problem with multiprocessing machines (ie, just about everything). You cannot just scan directory entries by index since those indicies can slide around while you are processing an entry. This means it is completely possible to process an entry twice, or skip entries completely.
I don't know if they have a good way around this problem, or if it even applies here.
In what way is ext2 unusable? I am currently using ext2 for this particular situation, and it seems to work ok. For example, here is a link that takes you to a file stored in a 60,000+ file directory. What am I missing here?
jeff@jab.org
OS/2 HPFS is designed around B-tree technology. OS/2 SMP servers seem to have no problem with it. Maybe the Mac just has a bad B-tree FS design.
Add encrytion and redundancy to the list.
undeletability = chattr -i
what about user-defined attributes (a la BeOS)?
on the mac, hard system crashes cause _much_ more damage on average than on winshit or especially linux. Granted, there have been shitloads of bugs in the mac fs code, but trees are still far more complicated than a simple hierarchy. If they are not implemented with 100% logical correctness with attention to methods of caching to minimize damage in case of a crash, trees can be an utter disaster.
... => If we cannot trim subject lines...) - check it out!
Don't come crying to me in 2 years when you want to know what all these "invalid sibling link" all over your HD are. I've been through it all. I'm not going this way.
oh, btw, on the PR for this: you should see the linux-kernel threads. Reiser's subject lines are so freaking long that some kind soul parodied one recently (If we cannot change filesystem semantics
Stephen Tweedie spoke at length about the new filesystems
at the UK Linux Dev conference last weekend.
The filesystems coming out now are pretty well tested
and it is unlikely that they will cause filesystem
corruption. There will certainly be bugs to be ironed out
but these code releases come AFTER an enormous amount
of debugging.
It is likely that a fully stable filesystem will be ready within 6 months.
Frank Xia remains an unsung hero of Linux developement. His filesystem overcame the size and field limitations of the Minix FS, and it also overcame the severe speed limitations of the Ext filesystem. He deserves a place in the Linux Hall of Fame. He bought time for Linux while a more sophisticated Ext2 filesystem was under development. Linux luminaries such as H.J. Lu swore by the nimble Xia system, and others might have abandonded Linux altogether if they would have had to use the problematic alternatives.
The files being served in the MindCraft benchmark are significantly larger than block size IIRC. Reiserfs shows its most dramatic performance advantage on sub-block-size files.
damm it.....could you people please read an intro book to IP. Simple question, what would the point of a license be if the copyright holder automatically kept all changes??
A license is specific allowance to do something, it extends the rights of the licensee. If a license gives the ability to alter someones code then this can be done. Those changes are under the copyright of the write, not the original. If the license specifically said any changes you make fall within my rights that would be different - this is the case with the NPL.
You cannot extend copyright in the way you are suggesting. You *can* do what you are suggeting wiht a license. If you want to comment on this very *important* area for Free Software (and you should want to) please, please, please read an intro book on IP/licenses
Obviously, you need some kind of concurrency control. This
can take the form of:
1. Lock the whole system during your transaction. Safe but
inefficient!
2. Lock each BTREE node as you use it. Got to watch out for
"deadly embraces", where two processes are waiting for what
the other has locked.
3. Optimistic concurrency control, something I read about
at least 15 years ago. You "check out" nodes as you do
the transaction, then "verify" when you're done. The
"verify" fails if another process modified one of the nodes
during your transaction, in which case you start over. I
don't know if this has ever been proven useful.
Not even close. It's a balanced tree that tries to minimize the block reads to find a specified code, specifically designed for filesystems. The balancing is accomplished by splitting and joining nodes to maintain the average number of children per node in certain limits. One node at a time is usually read from the disk and then the next is chosen on the basis of the that node, so having multiple keys in one node is faster than just 1 key per node as in a binary tree.
.. kN
like this (increasing order to the right, cant get less than to work...)
k1 k2 k3 k4
There's a child node between every key (and before the first and last keys) and the value of every key in that child node (suppose the node is between keys K1 and K2) is between K1 and K2. In a binary tree we can cut out 1/2 of the unsearched keys per node, in a B-tree we can cut out N/(N+1) keys. The rationale is that searching a node with multiple keys linearly is a lot faster than reading another node from disk.
In addition this fs uses a B*-tree which has some additional constraints (I can't remember exactly, was it that every node has to be 75% full at all times)...
Dear Mr AC (no, not you Natedog!):
It's a long paper, so just skip all that boring file system stuff, and go here.
--
--
=8^
Yeah...I realized that after I hit submit. Oh well.
>introducing B-trees into ext2
/usr/local/bin is a whole lot easier than /right/right/left/right/left/right/right/right.
Geez, I can barely remember where everything is
now. cd
cd
;)
Not really as the whole working set was only 75 MB and thus fitted completely in the cache. For an idea what happens when the working set is larger see the C't benchmarks (it could be though that RAID5 is extremely bad for NT but that's a pretty usual setup).
1) You can dsirtibute your _own_ code under any number of licenses you want. So you can distribute it under both the GPL and under other licenses for use in proprietary code.
2) Namesys wants money from distributers of proprietary Unixen for whom the GPL is unacceptable, I very much doubt they also expect money from distributers of free operating systems.
The main author has to get permission from everyone who submits patches in order to use them in the proprietary versions. Or he can rewrite them himself. neither are particularily hard. Peter L. Deutch has been doing this for years with Ghostscript.
but you got it wrong! its up up down down left right left right b a b a select start (omit the select on some games)
ahh nintendo...my ONLY memory of childhood
That's odd, I've been using it for over a month! At the moment I only use it for fairly small loopback filesystems (it's still awesomely fast). Once I get a better feeling for how stable and corruption-resistant it is, I'd like to use it for all my filesystems. /usr/doc (on Debian systems) would rock using reiserfs (it brings out the worst in ext2). I'd say the transition from ext2 -> reiserfs is almost as awesome as the transition from FAT16 -> ext2. Especially in terms of space efficiency.
fish and pipes
Last time it was discussed (that is, just some months ago), the answer was : in (almost ?) all cases where someone could wish for user-defined attributes, they can be emulated with directories, and directories have the advantage of not breaking existing applications.
That's another reason to use reiserfs btw.
I don't think either belong to the filesystem layer, what's wrong with encryption (loop) layer and raid ?
Ok, so if I count in the "ext2 with b-trees" as one future option for a file system, I now have XFS, Reiserfs and b-ext2 as an option.
Could someone clearly give a nonbiased look at what the good and bad points of these are?
I know I'm using a machine with a b-tree based file system right now and occasionally when it crashes, things can get very hairy. I'd take redundancy over speed any day, so I guess I'm for XFS?
"Although it is not true that all conservatives are stupid, it is true that most stupid people are conservative."
The code may be good, but the PR sucks. Even to a frenchman like myself, the language used in the press release sounds clumsy and amateurish.
Does it have compression? Compression built into the filesystem is a cool thing. What other features does it have?
support gun control: take guns from cops
ext2 with b-trees has been promised for some time, and when it arrives it will not be using modern tree technology. XFS for Linux is a press release, and they will be making it an XFS lite, and they aren't going to do the port themselves, they are going to let some university do the port for them, and they have legal issues with SVR4, and it just goes on. They say it is coming out soon, but they need changes in the buffer management code, and....
It is vaporware, which is not to say it won't ever be real, but.... we are shipping now, and we aren't using obsolete forms of B-trees. When they ship, we will be even farther ahead by then....
All that said, XFS is a very nice, respectable, mature technology.
You can't really use ext2 for this application, it becomes unusable for 10,000 file directories.
I am afraid you have little alternative to us for this particular application.
We do quite well at the create, delete, write, operations that dbench has a lot of.
We hope to do a lot more extensive benchmarking, but at the moment it seems that once they fix the networking code enough that the FS is the bottleneck, we will be able to help.
Actually, Andrew is named after both of CMU's founders, Andrew Carnegie and Andrew Mellon. (CMU = Carnegie-Mellon University)
I think the general model will be to use ext2 by default for a while, but ultimately reiserfs looks very interesting for most 'desktop' style usage of Linux (including servers for desktop systems), e.g. mail, news, reasonable size databases, etc.)
:)
XFS looks more appropriate for large databases with huge disk sizes and file sizes (and stringent time-to-recover requirements), as well as for huge multimedia files (which SGI of course is traditionally focused on).
I don't think a merger of reiserfs and XFS is practical or desirable. What might be very useful is a tool that takes performance logs (e.g. sar(1) output on disk I/O) and scans your filesystems, then recommends which files should be on which filesystems.
One more blue-sky idea is based on the observation that you have to move the file to a different part of the filesystem tree to change it from (say) ext2 to reiserfs. It would be very handy if some user-level daemons could automagically migrate files to different filesystems based on sizes and usage patterns - just as hierarchical storage systems move files from disk to tape etc, this would change the underlying implementation without the user having to worry about this.
However, this would require you to separate the filesystem namespace entirely from the actual underlying filesystem, on a per file basis perhaps, which introduces all sorts of complications. On a much coarser level of granularity, this is somewhat analogous to having a logical volume implemented on top of various physical disks. But this is probably a feature for Linux 5.1
cd /usr/local/bin is a whole lot easier than cd /right/right/left/right/left/right/right/right.
It dramatically decreases different technical details, doesn't it? Must be more user friendly then. Like mice with less buttons and parameters hidden behind multitudes of dialogs.
:P
[dicklamer: i like macs too]
I think, therefore thoughts exist. Ego is just an impression.
Quoth Namesys' technical introduction:
:P
This project is GPL'd, but I sell exceptions to the GPL to commercial OS vendors and file server vendors.
Seems like you'd have to derive your own version of GPL in order to not break the original by yourself - unless it already makes this possible. Well, I guess I should read it myself before making comments like this
An interesting situation will arise if reiserfs or major parts of it will be applied in Linus' kernel releases or optionally patched into distributions. Namesys' effort to fund itself with its GPL'd code might fail, if their product gets provided implicitly. Practically and hopefully this might be solved by RedHat et al striking a deal with Namesys. However it's a stretch of the spirit of GPL to assume a reward although the code is free. This kind of self-funding free development should be supported, but is it really? Or am I so mislead that I should just bury my nose to the dear old source of GPL?
Anyone shed some light on this fundability vs. GPL issue?
I think, therefore thoughts exist. Ego is just an impression.
My god, am I really already too lazy to read even my own writings?
On a side note, I found Namesys' technical introduction well written.
I think, therefore thoughts exist. Ego is just an impression.
1 - Well, bringing community changes from under GPL to the proprietary version sounds still like breaking the rules
2 - This sounds very reasonable. So they expect 'real' unices as paying customers. In a world somewhat fair, I can figure them getting funds from several big-time distributors, opensores or not.
I think, therefore thoughts exist. Ego is just an impression.
Answer to an implicit question:
Distributions can include the GPL version. No deals are needed at all. I have no idea where you got that idea from, since GPL is spelled all over it.
Question was not about deals or no deals, but where to make money from. That problem arises when they try to make a (/some) living out of their fs. But actually it's just that 'ages old' opensores business model: A good piece of code will be supported by those who find it useful. Even with cash, where that's appropriate.
Misunderstanding is natural considered the hazy nature of my wonderings above.
Code forking between the different licenses might become a problem. Then again, problems are just something to solve.
I think, therefore thoughts exist. Ego is just an impression.
I have about 1 million files, averaging about 5k apeice, on an IDE drive and being served by a web server. Some directories hold tens of thousands of files. The files are served from a low traffic web server; well under a gig per day. Is reiserfs better than ext2 for me?
:)
-rozzin.
Could you recommend such a book?
My Web Page
like many before him, he could hire a marketing firm, pay them to pick a slick and buzzword complient name, and then charge for it.
In OSS, you may not get as much $$ as those that charge for their work, but one of the main motivations is exposure. Just as Linux was named after Linus giving him credit for what he started, let Mr.Reiser have credit for what he has done. sheesh!
BTW - if you had gone to Mr.Reiser's web page and actually read it you would have seen that he addressed this at the bottom of the paper.
\forall code \in C, \frac{\Delta readability(code)}{\Delta t} < 0
Build a better mousetrap, Release it under two licenses: One open but non-profit, one closed but for profit. Just like qt does. You can make alot of friends, and you can afford to pay for lunch too.
Only problem is that you depending on the legal system to defend your interests, which could be an expensive and difficult task.
Anyone got any ideas about how to solve this?
There is nothing wrong with this. The copyright holder can release the code under multiple licenses separately... they can release it under the GPL (which is then good for everyone) and they can also release it under a different license to a specific party. What this means is that the specific party is free to make changes that they do not need to release under the GPL. However, only the copyright holder can do this.
There are other packages that are released under multiple licenses. Ghostscript is one... in fact, ghostscript is released under a more restrictive license first, and under the GPL later (with a delay of about a year, I think).
Anybody else can confirm this?
----------
In a real emergency, we would have all fled in terror, and you would not have been notified.
Disclaimer: I work for SGI. SGI created XFS.
I've been running XFS on my IRIX desktop since 1995. There is one fairly good reason to use XFS on the desktop, which I haven't seen mentioned: you never ever have to fsck.
K<bob>Papa Smurf aside, I always figured that AFS was developed at CMU, and named after Andrew Carnegie.
Just a guess, though.
I doubt it.. but would using this fs
have any dramatic impact on the 'MindShaft'
Benchmarking results?
No.. I"m not quite THAT hip to exactly
what this would accomplish.
I am curious though.
Friends don't let friends buy Compaq's. (Dell/Gateway... same same) You want a good computer? Build it yourself.
Ok, so we've all noted that peformance is awesome for 100 byte files. But check out his section on larger files too. The benchmarks of files all greater than 512k are also quite impressive:
Look at the rm -rf * comparison--
Ext2: 0:02.63
Reiserfs: 0:00.16
Ratio:16
Ok, so we've all noted that peformance is awesome for 100 byte files. But check out his section on larger files too. The benchmarks of files all greater than 512k are also quite impressive: .16 second rm -rf * time for 80+ megs in a series of subdirectories is pretty damn impressive.
Look at the rm -rf * comparison--
Ext2: 0:02.63
Reiserfs: 0:00.16
Ratio:16
Wow, now a new user logged in as root can wipe out his/her system 16 times faster than ever!
Seriously, though, a
--JZ
Check out the results for the 80/20 banded file set. A key result is that Reiserfs performs MUCH faster when large numbers of files are grouped in one directory. Copying all the files from one directory under Reiserfs was 12 times faster than the same operation under ext2. Scanning all files (if located in only one directory) was improved by a factor of 6. These results drop to ratios of 2 or 3 to 1 if the files are in many subdirectories. This uses a mixed set of both large an small files (about 50 megs' worth), but emphasizes the very smallest files (12,000 of the files are under 100 bytes). Those really small files provide reiserfs with the big edge.
However, its results are still quite good for medium-large files and the author admits that he's done virtually no optimizing for larger files. I suspect that, if Reiserfs catches on, we'll quickly see these optimizations in the next few months. Thus, I'd wait a little while and see how things pan out before putting it into use.
Does it address the other ext2 limitations? In particular, does it address the lack of metadata for new features such as:
1. access control lists
2. capabilities (keep them out of ELF -- yuck)
3. compression
4. undeletability
BTW perl is under two licenses also, the GPL and the rather less restrictive Artistic License. The GPL is somewhat viral in nature (which is the point of the GPL - so no surprise there), but that is not acceptable to everybody.
c't/heise Linux/NT benchmarks
It's good. Read it!
Summary:
- NT is faster in one unrealistic test.
- Linux is much faster in one somewhat unrealistic test (CGI on NT).
- Linux is much faster in a more realistic test (many files).
- Linux is slightly faster in all other tests.
I still wonder why the Linux guys accepted the unrealistic test for the PC Week benchmark ("Mindcraft 3") and didn't insist on additional realistic benchmarks.
Did I miss this one on the Slashdot front page?
Or weren't they posted?
If they weren't, please someone kick Rob...
Naturally, you wouldn't use something like this for multimedia streams, but I think the developer's point is that for a lot of applications *should* use lots of little files. Programmers often clump the data together into large files to avoid performance issues or to provide indexed retrieval (e.g. dbm).
I think there is a lot of application for something like this in applications that track lots of text files (document management, source code control, etc). Just think of how much simpler a lot of your tasks become. For one thing you get all the great UNIX file and text utilities like "find" and "grep" prebuilt and ready to use with your data. For security, you get creation and modification information, and can specify permissions on individual objects within a database like rows or fields, as opposed to entire collections like database tables.
I think a filesystem like this could become very popular in web applications, both for efficiency and because of ease of programming. I could easily see something like slashdot implemented on it.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
cd/up/down/left/right/b/a/select/start
"If one is really a superior person, the fact is likely to leak out without too much assistance" -- John Andrew Holmes
If you write a piece of code, you (or your boss) owns the copyright. If you own the copyright you can release it under whatever license you like. For example to the community under a GPL license, and to paying customers under a different license.
If reiserfs gets included in the kernel, and the kernel developers modify it, these modifications are applied to the GPL'd release, so fall under the GPL again. Reiser won't be able to include these patches in his commercial version (unless he explicitly asks for and gets permission).
Distributions can include the GPL version. No deals are needed at all. I have no idea where you got that idea from, since GPL is spelled all over it.
Anyway, just to reiterate, since you're not the only one who doesn't understand (I'm thinking of a certain BSD advocate who recently tried to crush the GPL using very false arguments): if you release your code under the GPL, the GPL only applies to that release. You are still the owner of your code, and you can do with it whatever you want. (But of course you can't retract the release itself.)
From the press release:
Reiserfs is GPL'd, with exceptions to the GPL available for sale.
I didn't quite get that bit. Is Namesys selling derivatives of GPL'd software with a commercial license? Isn't this supposed to be the big no-no in the GPL? I checked the website linked to in the press release but couldn't find any other mention of this. Does anyone know any details?
Some of the admited architectural weak points worry me, especially with large files. I'm worried Resierfs is a niche filesystem, great for systems comprised of small files, but not really industrial strength. However, I'm not acquainted with some of the finer points and numerous "common" optimizations he refers to, so I could be wrong.
Reiserfs and XFS both being released so close together might divide the rather small pool of Linux filesystem hackers who are qualified to merge all these ideas into something workable. Plus, I haven't grasped either to the extent that I can tell if they're able to be merged.
To some up, I don't know what I'm talking about, but I'm nervous nonetheless.
"There's so much left to know/ and I'm on the road to find out." -Cat Stevens