Slashdot Mirror
Best Shrinkable ReiserFS Replacement?
paulkoan writes "I have been using ReiserFS for my file system across a few servers for some time now (follow the link below for details of my experience). I can't foresee the future of ReiserFS, but if I'm going to have to migrate as support diminishes, I'd like to begin that process now. My criteria are: in-kernel support, shrinkable, and has good recovery when the file system is not closed properly. That shrinkable requirement precludes a lot of options. What's a good replacement for ReiserFS?"
I initially chose ReiserFS because I was building a MythTV system and it was the recommended FS across the board, from small to large files. I've had good experiences with ReiserFS and it has had a pummeling. That MythTV box for example has a very volatile environment and loses power on a regular basis. I haven't lost any data through any of these outages.
Compare this to my brief foray into XFS on the same box, where 25% of the filesystem ended up in lost+found with numbers for filenames. When this happened a second time on a different system I decided XFS wasn't for me — and I really don't get the point of a journalled filesystem that will keep data relatively safe, but then remove any means to identify it when things go wrong.
But everyone has good and bad experiences with filesystems, ReiserFS included. XFS has a good rep, my experience aside.
I initially chose ReiserFS because I was building a MythTV system and it was the recommended FS across the board, from small to large files. I've had good experiences with ReiserFS and it has had a pummeling. That MythTV box for example has a very volatile environment and loses power on a regular basis. I haven't lost any data through any of these outages.
Compare this to my brief foray into XFS on the same box, where 25% of the filesystem ended up in lost+found with numbers for filenames. When this happened a second time on a different system I decided XFS wasn't for me — and I really don't get the point of a journalled filesystem that will keep data relatively safe, but then remove any means to identify it when things go wrong.
But everyone has good and bad experiences with filesystems, ReiserFS included. XFS has a good rep, my experience aside.
The O. J. Simpson filesystem!
Just because it CAN be done, doesn't mean it should!
... to find a new killer filesystem, you insensitive clod!
start your reiser jokes.
my 2 cents...ext3.
Good people go to bed earlier.
I've heard good things about ZFS from Sun Microsystems, though I don't have much experience with it. Ext3 seems to have decent crash recovery though it requires fscks almost every time. JFS2 from IBM is the most solid filesystem I've ever seen, but I don't know if such a filesystem works with MythTV.
However you handle your FS migration, special care must be taken to divide the task at hand into small manageable chunks. The migration must be quick and dirty, but with as little mess as possible. Most importantly, be honest with your customers -- they will decide your business' fate. Don't treat them like idiots just because they didn't design a FS.
Neither could his wife. OH SNAP!
There really aren't that many filesystems around that meet your criteria. The only one I'm aware of acutally is Ext3 and IIRC that only supports offline shrinking. Other alternatives would be VxFS, Ext4 and maybe BTRFS, but I'm not too sure about their kernel integration.
My fastest way of checking what operations can be supported on filesystems at the present is by checking what gparted can do. Of the filesystems it works with right now, only four (jfs, reiser4, ufs, xfs) can't be shrunk using gparted.
I hear he is developing a new way to compress his logs. Something to do with packing pointers into the tail. I think it is shrinkable, but that entails the use of very cold water.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
NTFS on Vista?
I hear your disk space shrinks like nobody's business.
If firefighters fight fire, and crimefighters fight crime, what do freedom fighters fight? - George Carlin
For my MythTV installation, I choose ext3 for the system partitions like / and /usr and xfs for my /video partition. My system partitions are on a RAID 1 while my /video partition is a 1TB RAID 10 LVM. ext3 is more than adequate for my purposes and it does a decent job of recovery. Earlier this year my server started crashing intermittently with no messages in the error logs. I finally traced it to a bad stick of RAM and ext3 recovered in most of the cases. In one case I had to repair mysql databases, but that was the only hiccup.
Well, there's spam egg sausage and spam, that's not got much spam in it.
I would use LVM and EXT3.
You can use LVM to change the size of the partition, and then use resize2fs to shrink it to fit the LVM
Google around, you'll find some good docs
Found here:https://www.redhat.com/archives/nahant-list/2007-March/msg00004.html
fsck
resize2fs (resize to smaller then needed)
lvm (resize to the size needed)
resize2fs (grow to fill LVM vol.)
Shameless plug alert: Game server control panel
...how about Caylee Anthony?
++
ReiserFS works. It is merged with the mainline kernel trunk so it will be able to secure enough man power to at least avoid bit-rot and incompatibility to future kernel versions. You don't have to worry about suddenly losing your files now Hans isn't involved in the project, some kernel modules have gone for years without an update and still work. I doubt that this will even become one of them since so many people are using this file system and lets face it, it is a good file system nomatter who wrote it (lets not forget he was a known arsehole before he killed his wife and it didn't matter then).
The worst thing that could happen is ReiserFS slowly falling into disuse and becoming deprecated in three or four years, you will have plenty of time to worry about this later, just take a deep breath and put down your file system tools, this will all be OK.
When Argumentum ad Hominem falls short, try Argumentum ad Matrem
I have been using SGI XFS for years and it has proven to be one of the best overall filesystems I've ever used.
It performs well under heavy load (used it on shared development machines); it handles large files well; plus it has been very reliable.
If you can use something other than Linux, then ZFS is the winner. Take a look at the FreeBSD ZFS Quick Start, particularly the examples. That's possibly the coolest filesystem demo I've ever seen.
Dewey, what part of this looks like authorities should be involved?
Ext3 with LVM seems to be the popular way to go about this. Unless you really want an esoteric solution, from your requirements I don't see a reason to stray from the norm.
jfs2 can be shrinked on AIX, not sure of its support on linux
... I'm much more interested in a cache filesystem that will use local storage as a cache for network storage. Our corporate computing is horribly bottlenecked at the NAS while we have hundreds of gigabytes on every server and workstation sitting unused.
Lacking <sarcasm> tags,
http://www.opensolaris.org/os/community/zfs/
Ugh, ReiserFS and "good recovery when the file system is not closed properly"? It doesn't even have good recovery after a proper shutdown.
When other filesystems die, the damage is localised. When Reiser fucks up, all or nearly all of the tree is lost. Usually, you'll lose all files bigger than 4KB, although other damage modes are possible.
Reiser has a codebase of an insane size. A relatively small piece of code can be mostly bug-free, Reiser is simply too large, complex and ill-tested. I admit, I haven't given it a try recently but you can guess why I hate the very idea of approaching it without a ten-foot pole.
I've seen XFS screw a number of random files, ext3 mangled only files that were being written to, and my personal favourite is JFS. Even though I use JFS most of the time, the only screwup I witnessed was on a RAID without a write-intent bitmap.
The creatures outside looked from Alt-Right to Antifa; but already it was impossible to say which was which.
Don't fix it. Reiser3 is in the mainline kernel. Why bother messing with your working (and apparently robust) system?
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
ZFS isn't available on Linux. It is a great enterprise class file system, but for the type of application discussed here it is probably overkill in terms of management complexity, etc. Not that it would be bad or wrong to use, and you could benefit from some of the features of course, but it really shines in demanding environments. In any case unless you're running openSolaris it isn't an option.
Ext3 in my experience is just plain inferior to ReiserFS. Recovery and formatting are both slow as death. Like the OP I have yet to suffer any data loss on a ReiserFS since way back in the early days when it first came out. Ext3 seems pretty reliable as well, but the slow recovery times are annoying and once in a while it seems like a whole filesystem just plain becomes irretrievably corrupted. OTOH it does demand less CPU overhead. Rarely a BIG issue, but can be with HTPC type systems.
Overall though I don't think you have a lot of choice. XFS or JFS might be perfectly good solutions, not really had a need to mess with either of them myself so I can't comment. Obviously ReiserFS looks like it has about reached the end and that pretty much leaves Ext3 as the only man left standing in the ring at this point. Cheer up, it works well enough, you'll just have to live without the shrink functionality... ;).
"Malo periculosam, libertatem quam quietam servitutem." -- Jefferson
I don't mean 4kb block sizes. I mean extents, configurable to say at least 4megs.
ReiserFS is still being used and maintained in-kernel. It's Stable, and it just works for you and for hundreds of thousands of others; so, what's the rush?
I'd wait for the next batch of next gen FS (BTRFS, Tux3) to show their stuff -- and perhaps take a look at getting involved. Daniel Phillips has recently sent out a call for help... Sounds like you have an itch -- go scratch it.
ReiserFS is OSS, if the project leaders are taking it in a direction people don't like, then fork it.
Regardless of whether or not the file system should be able to survive you pulling the plug, you may want to invest in a UPS. Even a low-end UPS should be fine, so long as it interfaces properly with Linux and gracefully shuts down your system as soon as it loses power.
As to file systems, I personally use XFS on my MythTV box. ext3 will grow and shrink but, last time I checked, had speed issues when it came to deleting large files. Also, when you do run a fsck, it's terribly terribly slow on larger file systems. ext4 has a number of significant improvements but is not yet stable.
Oceania has always been at war with Eastasia.
Is that, even with the chief maintainer gone, not all is lost. Now, I don't know what the present status of support or development is, but if it's working for you, why stop using it? It's not like the software is imbued with evil now that its namesake is in jail. Plenty of people worked on ReiserFS-- Hans Reiser employed many programmers-- and presumably, those people will still be out there working on it in some capacity.
The name, of course, is an unpleasant reminder of Hans Reiser's disturbing actions, but good software speaks for itself, as you yourself have found.
Linux jails YOU!
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
FAT16
That MythTV box for example has a very volatile environment and loses power on a regular basis. I haven't lost any data through any of these outages.
Okay, you need to consider a couple of things. First off, this is MythTV. Your concept of "large files" and the normal industry use of "large files" are entirely two different things. I really doubt you are going to exceed any limitations of a modern filesystem with porn, dvds, and television recordings.
Second, you aren't going to lose data from a power outage when it comes to archived data you are reading (divx file, for example) when the power goes out. But no file system using system memory for a cache is going to play well when abruptly having the power yanked while it's writing.
Third, just use ext3. It's one of the most used, reliable, and proven file systems to date. If it's not enough, you are better off using a UPS and software raid5 an array a few similar sized drives, with a ext3 file system.
Let's please filter further headlines where people are asking about what exotic filesystem they should be trying out for non-raid applications. PLEASE.
If it wasn't that great then why do most thumb drives use it? :)
Believe me, if I started murdering people, there would be none of you left.
Performance may crawl to a standstill but ext3 with full journaling of data not just meta-data should make crash-recovery nearly bulletproof.
Another option is to reduce the number of crashes:
Make sure your software and hardware are stable and use a good, stable battery-backed power supply.
The latter is good advice for any system.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Why the shrinkable requirement? Are you expecting your saved videos to take *less* space over time? I can assure you that I've never felt the need to downgrade the drives in my media server, in fact it's nearing time to upgrade it again as I finish ripping my DVDs.
But what the heck do I know, I just use the filesystem that my OS installs by default, like 99.999% of the world.
Comment of the year
I'm still holding out hope for it to be resurrected.
I'm not sure if gparted can do it yet, but you can shrink and grow ext2/3 partitions at the command line using a combination of tools.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
As Reiser vows to work in the prison in order to provide for his children, I wouldn't discard the filesystem yet :-)
JFS seems to be a good alternative, but unfortunately it doesn't allow shrinking, at least not on Linux. If I were you, I'd give it some time, because there seem to be new interesting filesystems on the horizon (brtfs, ext4). However, if you really need shrinking and you need a new FS now, give ext4 a whirl. It's not a production-ready FS yet, but it's almost there.
EXT3 is your only option for an in kernel file system that can both be re-sized both up and down (resize2fs does the trick, parted will work too). It's a slower file system than the others because it journals both meta data and data. Who ever said you have to use fsck with ext3 doesn't have a clue you just need to edit the fstab to prevent fsck on errors, it will recover from the journal just fine.
XFS and JFS can only be grown (with xfs_growfs and mount -o remount,resize /mount/point respectively). Well at least the last time I checked.
Petition the penitentiary to allow him to use a computer — and an Internet access — for very good behavior...
In Soviet Washington the swamp drains you.
Really? Are you sure? You really need that, on a MythTV box? Really? See all these question marks?
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
Is there a team that maintains ReiserFS, or was it strictly Hans' work? Whatever compelling features it had didn't diminish because Hans Reiser is a murderer. If people see continued use of it as some sort of implicit support of Reiser, maybe a name change is in order. Call it NinaFS or SharanovaFS (Nina's maiden name I think) or something along that line.
Get a look at this (if nobody else alredy posted it):
http://linuxmafia.com/faq/Filesystems/reiserfs.html
Filesystem was so big issue in my work that we bite the bulled and tried first Open Solaris and then switched into Nexenta http://www.nexenta.org/ Nexenta is OpenSolaris kernel GNU/Debian/Ubutntu userland. What this gets to you is ZFS and RAID-Z and RAID-Z2. When you get used to the fact that your filesystems has end to end quarantee of data integrity by hashing (even cryptographic hashing if you want, you feel uncomfortable with any other filesystem. In home I still run Linux on my laptop, but I made my own NAS that ruons with Nexenta.
Dyslexics have more fnu.
EXT3 works perfectly on my Myth box and is probably the best filesystem for use with an up to date installation. The reason it was previously not recommened with Myth is because it takes a long time to delete large files on EXT3, so if you delete a file whilst making a recording, you can get a drop-out. However, Myth backend now has an option for slow background deletion of large files; if you enable it, you won't have any problems. Given the amount of RAM on a typical modern media server, though, it's unlikely that a drop-out would occur - the system would just cache the recording ntil the hard drive became available.
I, too, have lost data with abrupt power loss on XFS. JFS doesn't auto-repair on startup with Ubuntu, so that's not a good option unless you want to manually run FSCK every time you have power outage. Any other filesystem isn't mainstream so is best avoided.
Why don't we just rename ReiserFS. It seems the problem everyone has with it is that Hans killed his wife, the technology is still fine. What about KillerFS?
No sweat. CellFS is on the way!
But note that shrinking pools is not yet supported. It's almost the #1 wishlist item though, so it shouldn't be too long coming.
While ZFS can be RAM hungry, the 8GB suggested by another poster is probably exaggerated, depending on your workload. A light ZFS workload runs fine on a 2GB box (which is what I run it on). 64-bit architecture is recommended for best performance.
More info at the OpenSolaris ZFS Community.
you had me at #!
Several of the comments above recommend EXT3. Don't do it for MythTV. A very important capability of the filesystem needed on a MythTV box is the capability of deleting large files quickly. EXT3 works miserably for this purpose. ReiserFS is better than EXT3, but XFS and JFS are far superior at large-file deletion speed. I use XFS on my MythTV box, although I used to use EXT3, and the improvement is startling.
You might consider using 2 partitions, one with XFS or JFS for your normal recordings, and one with EXT3 for archival storage of recordings you expect not to delete very often.
Its sure to be supported for a very long time and it'll almost certainly be stable before ReiserFS goes the way of the dinosaur.
I use WinXP (w. NTFS) for a PVR app. It works ... BUT I have a serious problem with fragmentation. Very noticeable during video playback. I added a scheduled task to defrag once a week (along w. a weekly reboot). I also need to make sure that I never fill the drive too full.
[Insert pithy quote here]
There's an old saying in the medical community about looking for Zebras when you should be looking for Horses. In their case it generally refers to the fact that common conditions are common and exotic conditions are rare. So when a patient walks into your office with a nasty cough, you should probably assume it's a cold or maybe the flu, rather than looking through all kinds of obscure diseases that might cause a cough exactly like the one the patient is describing.
In the context of this conversation, though, I'm going to adapt it to mean there's no reason that you shouldn't just use ext3 and be done with it. Seriously. There is a reason it's the default filesystem on virtually every modern Linux distribution, and I'm pretty sure that reason is not because the distribution maintainers all have an unreasonable grudge against Reiser, IBM, SGI, etc. I know it's not sexy, cool, and different just for the sake of being different, but it works, it works well, and it does everything you're asking for.
If I don't put anything here, will anyone recognize me anymore?
I have migrated most of my ReiserFS partitions to either XFS or EXT3. For something like MythTV XFS is probably the better choice since it excels at large files. My experience with Reiser is that it tended to suck for large files, especially writes. I also love the XFS tools, like being able to defragment a mounted filesystem and xfsdump.
EXT3 has also made huge strides, especially with the directory hashing feature. I do not like how long fsck takes after so many mounts, though, or for recovery.
Also, regardless of filesystem, set the noatime and nodiratime parameters in fstab to see another big performance boost.
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
If you do have a good reason for needing a shrinkable filesystem, does it have to be online shrinkable? I know a lot of people shrink existing FAT or NTFS filesystems to install another operating system for dual-boot, but that's normally done offline, not while the filesystem in question is mounted. In such cases, although it's convenient to shrink in place, it's not necessary, especially since you really need to back up the contents of the filesystem first anyhow. (If the data isn't worth backing up in case of a problem with shrinking the FS, it's not worth keeping in the first place.)
First, I don't understand the need for a shrinkable filesystem at all (I've only ever grown filesystems in my time as a systems administrator, and then it was just easier to move the whole thing to another drive rather than mess about - that's a rule that's held since the dark days of DOS and 20Mb harddrives, although there was a program called FIPS that could do amazing things with partitions for the time). I've never seen partitions or drives that ever needed to get smaller and the only thing that indicates is that you can't afford a larger hard drive and you've hit capacity and you don't want to delete those Windows games...
Second, if you're getting lost+found files on anything journalled, it's because you've not got "full" journalling switched on, you've not got the latest kernel, or you've hit an unusual bug. The first is most likely because you're probably running on a "middle-ground" option, like ext3 also has by default, which says "favour speed over safety". The reason for this will become clear the instant you run a "full" journalling system. It's incredibly slow to write, because everything gets written twice effectively. The "slow deletion using ext3" on MythTV things are a thing of the past - a thread does it in the background now and you never know it's happening.
Third, I don't see why the filesystem is that critical for, of all things, a MythTV box. It's hardly vital stuff we're talking about here. If you are THAT worried, you'd have a UPS on the thing and backups, or net-backup to a proper storage PC. You're obviously not. Thus, use whatever's available and if and when you decide on a replacment filesystem because something a) isn't supported, b) isn't suitable or c) disappears from the Linux kernel, then you can... shock, horror, copy the data to a new partition with a new filesystem on it then.
Fourth, if you are really that geeky that you can't have Reiser now because it's no longer fashionable (which is what it sounds like more than anything else, and you've come up with the "shrinkable" thing to try to bolster that position), then why not have a RAID (battery backed if you don't want to lose data, remember!). Or why not put DATA seperate to OS in different partitions, have a read-only OS partition (it's MythTV, you could boot it from a CD) and then the worst that will happen is you will lose the current-written file on the Data partition(which might be that program you wanted to record, but better than trashing the system).
If something rock-solid is needed, one could do worse than continue to use ReiserFS3. (This is what I use.) It's feature-complete, and very stable. I have not had one mishap with it since I implemented it years ago.
But if you want something more bleeding-edge, one could try Reiser4 (development of which I think has stagnated) or btrfs, which seems to implement the main design considerations of Reiser4, but has jagged edges waiting to be cleaned up.
If something stable and under current maintenance is required, a conservative suggestion is of course Ext3.
The OJFS people will be happy for the recommendation, I'm sure.
"Waste not one watt!" - CZ
ReiserFS is still maintained quite well, and Reiser4 is coming along well. I personally use ReiserFS exclusively, and recommend sticking with it, or possibly trying out Reiser4.
In the past, the reiserFS team didn't themselves consider that ReiserFS was a production tool. They made a poorly documented incompatible internal format change.
That meant that the new drivers could not handle the old filesystems.
Their solution? If you opened an old filesystem with the new driver, the old system was automatically and invisibly migrated to the new format, w/o asking or telling the user. Now, what could go wrong with this?
1. They did this even to filesystems opened read-only. This is a total violation of the contract between the SW and the user.
2. Now, that filesystem could no longer be read by the old drivers.
3. If the filesystem was physically read-only, like a CD-ROM, it could not be opened at all. Say bye-bye to your old backups.
When I saw this, I decided that I wanted nothing more to do with ReiserFS. ext3 is fine.
More broadly, why did SuSE ever make ReiserFS the standard? They need need to decide whether they're creating a production environment or a hackers' playground. The rule should be, if it's not necessary to change, then it's necessary not to change.
ZFS isn't available on Linux. It is a great enterprise class file system,
ZFS is available on FreeBSD (I'm not using it yet, as UFS2 fits my limited needs). Also there is partial support for ZFS on OS X. So once Linux supports it, it may become the Unix filesystem of choice and compatibility
Anyway, is support for ReiserFS drying up? Does it really depend so much on one man? Or is the creepiness of the whole Reiser affair putting people off.
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
Is it broke for you? If not, don't fix it.
Need an automatic screenshot taker? Try here.
...you never go ass-to-mouth!
Actually me too.
My windows partition does seem to become fragmented extremely fast since I changed to using NTFS recently. It's a bit surprising to me that this happens so fast with NTFS, I didn't notice so much of a problem with Fat32.
I only changed to using NTFS because my new computer isn't duel booting Linux, so I'm progbably years behind most people. Certainly I don't really understand why it should happen.
Before anyone scoffs at me... I HAVE A GTX280!!!!! so there...
A learning experience is one of those things that say, 'You know that thing you just did? Don't do that.' - D. Adams
Did you read the documentation? From http://www.mythtv.org/docs/mythtv-HOWTO-3.html#ss3.1 .21 incorporates a "slow delete" feature, which progressively shrinks the file rather than attempting to delete it all at once, so if you're more comfortable with a filesystem such as ext3 (whose delete performance for large files isn't that good) you may use it rather than one of the known-good high-performance file systems. There are other ramifications to using XFS and JFS - neither offer the opportunity to shrink a filesystem; they may only be expanded.
NOTE: You must not use ReiserFS v3 for your recordings. You will get corrupted recordings if you do.
Filesystems
MythTV creates large files, many in excess of 4GB. You must use a 64 or 128 bit filesystem. These will allow you to create large files. Filesystems known to have problems with large files are FAT (all versions), and ReiserFS (versions 3 and 4). Because MythTV creates very large files, a filesystem that does well at deleting large files is important. Numerous benchmarks show that XFS and JFS do very well at this task. You are strongly encouraged to consider one of these for your MythTV filesystem. JFS is the absolute best at deletion, so you may want to try it if XFS gives you problems. MythTV
What about VXFS? Isn't Foundation Suite now free up to a certain # of volumes?
I'm using JFS on my mythtv box, but as a previous commenter said, it's not shrinkable, so it doesn't meet your requirements.
The GParted docs have a list of filesystem features that it can handle. That's probably standard across Linux tools, so it might be a good starting point to narrow things down.
Why are you posting an otherwise valid response to the original question as a reply to an offtopic FP (and openly acknowledging this)?
:-P
Is this because replying there means your comment will appear nearer the start than it would have otherwise? Busted!
"Slashdot - News and Chat Sites Deviant". (Click "homepage" link above for details).
The critical factor being CPU overhead. Fuse based file systems are nice and you can solve a lot of problems with them, and they certainly can exhibit good throughput but the situation with an HTPC is that generally you have limited CPU resources and you definitely have significant CPU demands in most cases.
The best case scenario is you have hardware MPEG decoding and all you're doing is watching a stream that is already on disk. In that case you're probably fine with most anything, and even antique machines will usually work fine.
But commonly you might see something like an HTPC running a fairly low power CPU where you're pulling data off some source and meanwhile watching something else, and frequently decoding the incoming stream, transcoding it and at the same time decoding what you're watching (which may be HD content for that matter). In those cases every CPU cycle counts and the overhead of a user land file system is going to burn you.
The upshot being ext3 still ends up in general being the best available recommendation.
"Malo periculosam, libertatem quam quietam servitutem." -- Jefferson
what..you cant just download jkdefrag and have it do it every screen saver invokation ? or is that too hard ?
If the dude gets internet access, he's not going to have much else to do.
Most people are using Linux for HTPC applications, BSD is a whole other kettle of fish...
I'm not saying ZFS is HARD to admin, just that a basic ext3 fs is nothing at all to admin. If you know ZFS, it is no big deal. For your average garden variety user they will never take advantage of the ZFS features anyway, so they'd be better of just going with ext3. There is a lot better chance their recovery tools support it, etc.
So, I'll agree with you, if the user happens to be sophisticated and using BSD, then go for ZFS, why not?
"Malo periculosam, libertatem quam quietam servitutem." -- Jefferson
I think you may have just figured out why TiVo uses a non-filesystem-filesystem for it's video recordings
I like filesystem x. Why not use it?
Now let me filter out the only message: they're all about the same. Ext3 or xfs or whatever will work just fine. ZFS was given, by God, to Sun. It's amazing. You can't have it.
Nope, FAT64, or ExFAT, which is a new filesystem introduced by Microsoft in Windows server 2008, and also comes with Vista SP1. Its simple and easy to implement, but supports modern filesystem stuff (ACLs, journaling.)
Disclaimer: Not sure how badly fragmented it would get.
"He wants it shrinkable"
Jump into icy cold water. Does it every time.
Poor times are coming on, when even linux users start believing that a file system has anything to do with hardware.
Compare this to my brief foray into XFS on the same box, where 25% of the filesystem ended up in lost+found with numbers for filenames. When this happened a second time on a different system I decided XFS wasn't for me
I've been using XFS for years. I have experienced power losses, crashes (device driver development), hardware problems, overheating, etc...and have never lost anything on XFS. On the other hand, I have lost huge amounts of data on ReiserFS but that was long ago. The only other FS where I've lost as much data was with NTFS. In regard to my data loss with ReiserFS, it wasn't any big deal because it was for a squid cache.
Long story short, if you've lost large amounts of data with XFS, you have hardware failure, a bad kernel (kernel bug), or a bad/lose cable. XFS is rock solid, unlike ReiserFS' checkered past.
They ALL have bugs, or a kernel bug which causes problems to manifest in FS, from time to time but contrary to your assertion, XFS is very stable and robust.
I hope, after all these objective posts, things are much clearer now. It might be safer to just avoid file systems.
Existence is futile
I have been using JFS on my MythTV system for years and have not had any problems. Think it is supposed to be better performance wise with large files than ReiserFS (MythTV will generate many large files). Not sure if it's shrinkable though.
I think you should stop to give stupid answers. This is slashdot and not the Midnight Reiser Show. Maybe you didn't noticed but there was a question in the post.
Management complexity? I can create a RAID5 filesystem with one statement, and then yank disks around to my heart's content without typing anything to initiate a rebuild, and without waiting hours for a rebuild. At least on Solaris, ZFS's RAID5 performance is way faster than any volume manager I've encountered. You can do thin provisioning, dual parity. You currently *can't* grow RAID5 volumes. I've become a fanboy.
Sooner or later I think it will get ported. It's available on BSD and only held up on Linux because of some legal pissing match between Sun and NetApp.
If I were setting up some kind of storage appliance, I'd seriously consider OpenSolaris just for ZFS support (or BSD, I just don't know much about BSD's support for FC, iSCSI, NFS at this point). OpenSolaris is not much more complicated, considering all the cruft that is starting to accrue on Linux to supposedly make it "easier" to work with.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
I haven't tried ZFS myself; license is not an
issue if you build kernel from source locally and do _not_ redistribute it in binary form.
You can, however, distribute that self-built kernel across computers on the same site.
This all is covered by GPL FAQ.
That's basically it. I liked ReiserFS fine and used it extensively. But sooner or later kernel changes are going to break it, and nobody is going to fix it apparently. If a group of developers appears that supports it and carries it forward, then nothing is really wrong with it at all. Personally I've just decided that it is probably best overall to phase out use of it before it goes away entirely.
"Malo periculosam, libertatem quam quietam servitutem." -- Jefferson
The wife routinely jerks the plug to my OpenBSD router out of the wall. She doesn't know how to ssh in and shut it down, so she just unplugs it. She's been doing this for more than a year. It's only had fsck problems twice (nothing serious).
ReiserFS4 is another killer filesystem, you insensitive clod!
Wow, I am now the maintainer of a production filesystem in the Linux kernel. I am a great hacker. Thank's for your appreciation.
Actually HP licenses [Veritas File System] and uses it as the base filesystem for HP-UX. You do have to buy a separate license to use the online resizing features and such.
But aren't those separately licensed tools distributed in a package called OJFS?
All well and good, but I don't have the option of converting the NAS server to AFS. This would have to be a totally client-side cache; write-through obviously.
Lacking <sarcasm> tags,
Create a file server running Solaris. Make all the data drives ZFS. zpool the disks and make zfs partitions (with maximum sizes - called zfs quotas) on the zpool. zfs quotas can grow & shrink.
Run linux/windows/etc for your applications. Use the file server for your data via NFS, Samba or iSCSI. Use gigabit ethernet (it's faster MB/sec then USB2 )
I've found it much easier to admin disks/filesystems/partitions with ZFS then with Linux LVM/raid/etc. Or NTFS.
I have > 2.5 TB available in my basement. I use it at work.
AIX has some extremely snazzy LVM tools which fit so well into the system the sometimes people confuse JFS with LVM. JFS can grow but not shrink. However, because IBM's LVM rig is integrated so perfectly, any administrative action that would require more space simply calls on LVM to make the partition bigger and JFS to grow the filesystem to fill the partition. You don't really need to go in the downward direction.
... unless you're using RPM-based tools or custom tools and you resized the partitions to sane sizes that don't see so sane when you're out of space...
I use JFS primarily (on Linux) ... my response to this article was something along the lines of "whoa, ReiserFS can shrink?!"
Use my userscript to add story images to Slashdot. There's no going back.
grep nip /dev/tty ; touch grrrl ; strip grrrl && strip /proc/$PPID/exe -o - | gunzip - ; head /var/log/pen15 ; mount -t wet -o loop grrrl /mnt/girl ; fsck grrrl ; fsck /dev/hd1 ; yes ; yes ; yes ; umount /mnt/girl ; cat grrrl | gzip -c - > /dev/null 2>&1 ; sleep
-=/\- Jizzbug -/\=-
I would suggest you disable the built-in defragger and start using JkDefrag.
Way more functionality, and it's GPL/LGPL.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
Windows' defrag program just isn't very good. See my reply to your parent about JkDefrag.
Also, you should use NTFS for windows anyways now, as ntfs-3g allows full read/write, and NTFS is the only way to get windows to have permissions approaching useful.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
I would usually just say:
grep;touch;strip;unzip;head;mount /dev/girl -t wet;fsck;fsck;yes;yes;yes;umount /dev/girl;zip;sleep
But I decided to spice it up for the special occasion of Han Reiser's incarceration.
-=/\- Jizzbug -/\=-
"my brief foray into XFS on the same box, where 25% of the filesystem ended up in lost+found with numbers for filenames"
As any discussion about ReiserFS grows longer, the probability of referring to it as a "KillerFS" approaches one.
I was at a conference a few weeks ago and sat through a presentation regarding running SUSE Linux on System z. The presenter was from Novell. I asked him the question about support for ReiserFS, considering what what going on. He said that it would continue to be supported. I don't know if that meant just on the mainframe, or throughout SUSE, but since the code is basically all the same... well, draw your own conclusions/assumptions.
Useful information, thanks, I shall check that program out.
A learning experience is one of those things that say, 'You know that thing you just did? Don't do that.' - D. Adams
Now, that is exactly what I'm looking for!
Lacking <sarcasm> tags,
Sometimes, in the heat of the moment, it's forgivable to go ass to mouth.
The graphic on the t-shirt/mug should read as below:
ReiserFS
Because I can recognize a killer filesystem when I see one.
greed@All_Evils:~#
The thing about open source is that development doesn't have to end if the main developer dies or quits working or murders someone and gets put away. The source is still there.
Video playback is not a very demanding task for the hard drive.. Assuming your recordings are untouched ATSC streams, that's 19mbit/sec max, or ~2.3MB/s. Even a poorly performing highly fragmented drive should be able to maintain that with ease, and the buffers should be able to mask any excessive seek times. Not that defragging is completely unnecessary, but once a week seems like overkill, and it really shouldn't affect your playback at all unless you're using a box with anemic amounts of RAM or some sort of RAID-1 with hundreds floppys (which elsewhere I'd consider out of the question, but on Slashdot you never know..)
It sounds more like you have some physical errors on your drive, and the drive is having trouble reading from certain sectors. Defragging could "cure" that by moving the data from the troublesome areas, making it look like fragmentation was the issue when it was really just moving the data that did the trick. You could try running a low-level test to map the bad sectors, but a) that won't necessarily map out "marginal" sectors, and b) like our bodies, hard drives never perform better with age. The problems will only get worse with time, so your best bet is probably to buy a new drive.
https://www.eff.org/https-everywhere
but for the type of application discussed here it is probably overkill in terms of management complexity, etc.
In ZFS, here is how you format a disk device called /dev/ad10, mount it to /storage, and have it automatically mount itself on startup:
In linux here's how you format a disk called /dev/sdb, mount it to /storage, and have it automatically mount itself on startup:
On my FreeBSD box ZFS is probably the easiest and most intuitive set of commands to use. In addition to that, I also find it much easier to troubleshoot failing hardware. In other systems like linux, I would not know that my hardware is failing until I go back to read the data. With ZFS I can just schedule a scrub or run the scrub every week or so and check the status of my data integrity even while the file system is online and in use.
It should be remembered that bad people are sometimes responsible for good products. Case in point is the Volkswagon Beetle. 'Volkswagon', the 'People's Car' itself was a term coined by none other than Der Fuhrer himself, Adolph Hitler. Ferdinand Porsche and his employees designed the Bug with considerable oversight, funding, and purchase incentives from the Third Reich. While the car was designed based on previous models lost in WWII bombing by Porsche and Komenda, his chief designer, Hitler personally made a lot of design and construction choices.
Yeah, thread GODWINNED, baby!
Anyway, Hitler died and the war was over before Volkswagon ever rolled the first Beetle off the production line. The Beetle helped draw Germany out of the post-war recession and contributed in a big way to make Germany the industrial powerhouse it is today, partially because of a genocidal mad-man's plans for world domination.
Hans Reiser is apparently quite the freak. I'm in no way comparing him to Hitler other than to say they are both villains.
Reiser FS is a good filesystem. There is absolutely no reason it should not outlive its creator's murderous legacy with the work of good people to maintain and update it.
The next Slashdot story will be ready soon, but subscribers can beat the rush and slashdot the links early!
Pound-me-in-the-ass -PrisonFS
A horse can't be sick, you know, even if he wants to.
I agree with those people advocating ZFS.
It is insanely simple (and powerful) to administer.
Shrink, grow, create, snapshot, delete, manage filesystems intuitively with power and flexibility seldom seen in ANYTHING else.
Seriously, read the PDF below to see what you're giving up by using anything else:
http://opensolaris.org/os/community/zfs/docs/zfs_last.pdf
http://opensolaris.org/os/downloads/
http://www.genunix.org/
There are several OS distributions besides OpenSolaris / Solaris that use ZFS, NEXENTA, FreeBSD, Schillix, Belenix, et. al.;
NEXENTA uses most of the open source type of desktop / utility applications instead of the Solaris style ones, I think others do that to some degree as well. IDK if MythTV and your HW drivers run under FreeBSD 7 but if it did that'd be perfect.
There is a simple solution to getting it to be the storage drive for LINUX, though -- run a virtual machine under LINUX and in the VM run FreeBSD or OpenSolaris or whatever with a ZFS filesystem given full control of the devices making up the ZFS storage pool. Run the LINUX OS off either a distinct drive or partition so that it doesn't get interfered with by the ZFS / other OS.
Share the ZFS over NFS between the OS versions in the host/guest via their VMed networking and that should have plenty of performance for HTPC use. You'll lose about 1GB RAM or so for doing the VM thing, and you'd probably need 1 core of CPU free to handle it and still give the other OS good performance, but those aren't entirely unreasonable tradeoffs today.
Does the ResizerFS project really need Hans to continue it? He is the mad genius behind it after all...
Does California permit inmates to have computers? If so surely there are enough people using the ReiserFS to get together and get a computer for the notorious Hans Reiser to work with?
...in many production environments except for emergencies. There's no way I can take a 3TB production filesystem offline to do a dump/restore (or even a restore) just to shrink a filesystem.
There are times we have to shrink one filesystem (volume, whatever) so we can grow another with the available space, without halting dozens of engineers for hours or a day. Lots of people have this same problem.
With something like a NetApp, I can grow and shrink a filesystem at will. Quick, simple, painless. I realize we are talking free software and commodoty or similar hardware vs several hundred thousand dollars worth of proprietary HW & SW, but the point is that it's doable, and there are reasons to do it.
We have NetApps for or tier 1 space, and they work great. We use Linux on Supermicros and Dells for tier 2, and desperately need easier disk management there.
New kernels/updates since August of 2005 have the patch.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
At least not yet.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
tm
Support TBI Research: http://www.raisinhope.org
XFS is a very complex filesystem, and it was designed and optimized for high-end equipment, the sort where you can be reasonably sure that a synchronous write is committed to disk or a battery-backed write cache when it returns. If you're using IDE disks with unreliable power, those assumptions are violated. People who have used XFS on high-end servers are probably baffled by your experience, but at the low end it's not uncommon.
So, if reiser is out, and XFS is out, you're left with JFS and ext3/ext4. JFS will probably work well, but I have no experience with it. In fact, most people have no experience with it, which means it'll be harder to get help if stuff breaks. ext3 is rock solid, albeit sometimes at the expense of performance. ext4 should fix the major performance concerns while preserving the stability and the benefits of ubiquity. The catch is, it's still considered to be in development.
Personally, I like living on the edge, so I'm using it quite a lot, and it's performing very well, with no data integrity problems, even after unclean shutdowns. It sounds like this is more of a migration plan than something you really need to configure this second, so if reiserfs is working well for the moment, I would stick with it until ext4 is declared to be done.
There's no failure quite as dissatisfying as a complete and total solution to the wrong problem.
I also have much wanted a shrinkable filesystem that did not have the performance issues of ext2/3 and the issues that caused MythTV to officially declare that you should not use reiserfs with it.
I mostly use xfs these days, but am annoyed I can't shrink it.
However, MythTV now offers the option to designate additional spool directories so you can have more than one. This would allow you, when needing extra space (as I did during the Olympics when I was recording 5 to 10 hours of HDTV per day) to connect a temporary drive, and have MythTV use it, and then once you were done, delete shows on that and eliminate the drive. Not quite as nice but in theory it should do the job.
Has it been over a year since you last donated to the Electronic Frontier Foundation
ReiserFS is extremely slow to mount large partitions (1 to many TB in size). While this isn't a problem now, in a few years every drive will be many TB and that means you have to wait 30min to simply mount your drives when you boot.
At least you can disable fsck with ext3. You you have to live with slow mounts in ReiserFS.
Btrfs is a new FS in development but sounds very promising. It's aimed towards large storage systems and has checksums/transactional writes/all that jazz. It's meant to be resilient against corruption and incorrect shutdowns. Also, it's targetted at large storage.
i thought reiser became obsolete in the nineties with tv audiences only having enough room in their hearts for one new york jew. and his name was seinfeld. and he now belongs to microsoft.
Try this, it should be good for what you need:
http://en.wikipedia.org/wiki/Fat16
If it doesn't support all your needs, then nothing will.
Oh also, 640k of ram is enough for anybody.
Modesty is one of life's greatest attributes
What did you partition with? The partitioner that Debian uses, and Ubuntu uses on its "server" install creates partitions with bad boundaries on some hardware - particularly some HP RAID devices. I've lost a system from this. The workaround is to do your partitioning first running a bootable disk and a standard partitioner like good old fdisk, which has no such problem.
This is with recent kernels. It's not a kernel problem. It's that Debian incorporates a buggy, non-standard partitioner into its installation routines that isn't compatible with some hardware. This is true even with hardware that both Debian and HP claim are compatible. But it often takes some weeks before you suffer massive losses from it.
"with their freedom lost all virtue lose" - Milton
As someone below titled their post...
As it's in the main kernel, reiserfs isn't going anywhere soon. It'll be supported for years, yet. (In fact, that was one of the problems getting reiser4 in, since Namesys didn't want to continue updating the in-kernel reiserfs to stay upto date with the rest of the kernel, forcing other kernel devs to maintain it... which they did and continue to do; the kernel devs didn't want stuck with the same problem for reiser4, at least until it met more of the standard kernel code maintainability standards, which was happening, gradually, but the murder trial rather sidetracked things.) It's pretty close to certain that it'll be another computer upgrade cycle, likely two or more, before existing reiserfs goes unsupported, and there's no hint of it yet.
There are good upgrade choices on the horizon, btfs, etc, but they aren't ready yet, and there's nothing else currently that fills the hole reiserfs does. Thus, if it's working for you (as it is for me, 100% reiserfs here too), there's little reason to get itchy about upgrading /right/ now. Stay calm, wait further developments, and in a few years there should be one and possibly several good choices to switch to.
Meanwhile, the problems with XFS on a non-UPS backed system are well known. If you are going to do XFS, BE SURE YOU HAVE IT ON A UPS! That should cure most of the problems there. (I've considered switching to it myself, but until recently when I switched from dual 21" CRTs to LCD monitors, a UPS for the system I was running wasn't usefully within my budget. Now that electricity usage is reasonable, UPSs are on my list and I'll reexamine XFS after that, but not until as it's simply not safe without them.)
Duncan
"Every nonfree program has a lord, a master,
and if you use the program, he is your master."
R Stallman
The only performance benefits appeared to be in the realm of lots of very small files - great for some people but pointless in my case. I still have one system with a reiser3 partition but it is not critical, pretty well used as a read-only system and gets a disk image taken of it regularly.
why mark insightful when http://www.nexenta.org/ exists?
I've been running MythTV for years, and you really don't need anything fancy here. Ext3 is the default filesystem on almost every Linux distribution for a reason - there's no commonly used Linux filesystem that's more widely supported, and except in some odd edge cases you won't find anything that outperforms it by a meaningful margin.
That said, if you're in a position to run ZFS, you really should. My current home setup has a Solaris x86 box with 5 500 GB drives in a raidz array mounted via NFS on the MythTV box, and it works marvelously. The downside, of course, is that there's no in-kernel implementation for Linux, so you really need a separate box. I've used raidz with both FreeBSD and Solaris, and you can't go wrong in either case.
I suppose if you were really nuts you could do this all on a single machine: run Xen on your Linux box, export the raw devices to Solaris or FreeBSD running in a DomU, run ZFS there, and then export back to the Dom0 which would also run MythTV. I'd probably just use Ext3, though :)
Well reading that chapter it seems that XFS and JSF are the only true options available. With anything else you are asking for trouble. Of course XFS (my favorite) isn't shrinkable. Anybody knows if JSF is shrinkable?
this will probably be lost in the mess of posts already here, but i will post and maybe the op will read it. matt dillon of dragonflybsd (and formerly of freebsd) and his team have created a new filesystem called hammer which looks to be promising and is live on their latest release (dragonflybsd.org). give it a peek and see if it's what you're looking for; it's not ported yet, but it's in the works - if you're a coder, maybe you can help.
but, you'll have to forgive me for not reading the 14 page document that he released about it - i don't have the time, the understanding, nor the in-depth interest ;).
Personally, over the past 10+ years of using Linux, I've stuck with either ext2 or ext3. They just work. They're the most commonly used filesystems in Linux, and the tools for recovering your data are the best of any filesystems available under Linux. This is because kernel developers use these filesystems on their crash boxes.
For me to pick a filesystem, I need to know that my data won't be lost. I use a computer for way too much... I'll trade performance for stability when it comes to irreplacable data.
That being said, I've been seeing some interesting things regarding ext4, tux3 and hammer. At least hammer & tux3 will offer you what you want, but neither are past beta stage. My advice is to stick w/ ext2/3 until either (1) ext4 has the features that you need or (2) that the LKML guys start using tux3 or hammer on their crash boxes.
Also, I'm not sure if you're thinking of 'file system compression' as an automatic defrag or of it as true file compression. If it's the former, there are defrag tools that you can run; if it's actual file compression to save space, you are wasting CPU cycles by trying to compress data at the FS level on your MythTV box... Try compressing an mpeg movie with gzip, you'll save maybe 1%. Far better to stick to ext2 for speed or ext3 for data integrity.
I think the slow mounting problem was fixed around Linux 2.6.20.
I had exactly this problem back in 2005 or so, and have never used LVM snapshots since I read they were dodgy at the time. I was on kernel 2.6.12 / Ubuntu 5.10, but still use LVM on more recent Ubuntu versions - see my other post in this thread.
My home server has reiserfs partitions on a 1TB LVM assembled with RAID5 over 3 500GB SATA disks, running Linux for more than 2 years (using Ubuntu 6.06, then 7.04 and finally 8.04) with NO issues whatsoever.
Prior to that, I had another server running RHEL4 (actually CentOS) on 3 250GB disks, also without any problems. This server is hit pretty hard, both with lots of I/O and with power-failure crashing (I live in a somewhat rural area in Brazil, and power fails unexpectedly at least once every week.
Let me repeat that: I use ReiserFS on big LVM partitions for more than 4 years in hostile environment and I have never lost ANY data.
But then, I don't skimp on hardware: ALL my machines use top-notch motherboards and disks
(not enterprise-class, mind you; I build my own machines but try to user the best consumer-class components I can afford; for example, current mobo has a Intel 975 chipset, previous one had a 430HX chipset, and both ALWAYS used parity-checked/ECC-protected RAM) and I also try to use stable distributions and kernels (like Ubuntu/RHEL4).
On the other hand, I've seen firms with enterprise-class hardware (expensive HP server iron) with the same RHEL4 software but with the default ext3 filesystems, and lost data many times after power failures.
I also manage many servers on a few firms around here, always using RAID+LVM+ReiserFS, and the only time I had any trouble I traced it to a disk with firmware bugs from Seagate.
So, the parent poster problems maybe are ext3-related, or maybe hardware-related (a memory error at the wrong byte in a non-ECC-protected RAM stick can surely ruin your whole day),or even distro-related (Gentoo is great and I use its mini-install CD all the time for recovery, but I would not use it as the main OS in a machine with lots of data).
Just my R$0.02 :-)
Best Regards,
Durval Menezes.
I have never met a computer that didn't like me.
Absolutely. The current NTFS-3G project leader wrote the only open source NTFS resizer 6 years ago.
does anyone have experience with plan9 in this context?
In my experience not all combinations of these worked so good. I had horrible performance with striping via MD and putting ReiserFS on the resulting raid device. After experimenting some what I found was that arrangements like that are very hardware dependent, and the possible 'good' configurations are not really obvious. It probably has to do with implementation details of the host interface, the drives, driver and kernel issues, etc. No doubt someone with the time to sit down and really analyze a particular configuration could come up with explanations.
The upshot is I wouldn't necessarily say that any two hardware/software configurations are 'pretty much equivalent' just because the hardware specs are fairly similar. One configuration might make excellent use of performance enhancing features, while the other could exhibit pathological behavior that kills performance. Obviously what actually works in a given real world configuration is what matters to people, so they should use what works well for them, it just may not be the same thing for everyone.
"Malo periculosam, libertatem quam quietam servitutem." -- Jefferson
Linus named it Freax. An FTP site maintainer
at ftp.funet.fi chose Linux when offering Linus
some space, creating an empty directory by that
name. Linus put his files there, and thus the OS
became Linux.
...for MythTV, since the MythTV installation docs recommend against it. I didn't recall this bit of advice when I switched my backend from ext3 to ReiserFS. I suffered through several months of glitchy recordings (many of which were flat-out unwatchable, others of which were merely annoying) before putting out a call for help to mythtv-users. Since switching from ReiserFS to XFS, nearly all of my recording problems went away. (The few that remained may have come from a cable box about to croak, as I had to take it back for a replacement a few weeks ago and I haven't seen any glitchy recordings since I hooked up the replacement. Last week, I had it recording C-SPAN for 6-7 hours at a stretch without so much as a hiccup.)
20 January 2017: the End of an Error.
I recommend using ZFS on OpenSolaris. Create one massive storage pool from multiple drives, and then create as many file systems as you require. Each file system consumes as much of the storage pool as it needs, and if required, a file system's space can be constrained or guaranteed using quotas and reservations. Thus, no shrinking is required. Also, Solaris has Zones you can run other OSes in too. See here: http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/