Slashdot Mirror
A Good Filesystem for Storing Large Binaries?
jZnat asks: "I own hundreds of gigabytes of binary data, usually backed up from other mediums such as CDs and DVDs. However, I cannot figure out which filesystem would be best for storing all this reliably. What I'm looking for is a WORM-optimized FS that also has good journaling methods to prevent data loss due to some natural disaster while data is being shifted around. Trying something new for once, I tried using SGI's XFS due to its promising details, but I was met with countless IO errors after trying to write large amounts of data to it. I feel that Ext3 is not optimal for this; ReiserFS is too slow when it comes to reading large data files; and Reiser4 isn't mature enough to entrust my digital assets to. What filesystem would be most appropriate for these needs?"
Google made a filesystem for exactly that purpose: storing HUGE files highly reliably. OK, so it's not publically available, but it's still perfect for you (other than that).
journaling methods to prevent data loss due to some natural disaster while data is being shifted around
Journalling doesn't do this!. Journalling helps reduce file system corruption in the event of a catastrophic failure while modifying the file system - ie, it's possible to bring it back to the last clean state before it crashed - journalling does not prevent data loss. You might say "well filesystem corruption and data loss are the same", but they are not. If the filesystem is corrupted, the data is not lost. It just becomes not easily retreivable. If the data is lost then it becomes entirely irretreivable.
I tried using SGI's XFS due to its promising details, but I was met with countless IO errors
Have you considered your hardware is shit? I use XFS on terabytes of raided disks and have been for more years than I remember... 5 or so? I don't see any I/O errors. XFS is very reliable and I trust it with my data.
I feel that Ext3 is not optimal for this
Well not all of your post was dumb!
ReiserFS is too slow when it comes to reading large data files
How is it slow? It takes a few microseconds longer to access the first data sector because it does some extra processing first? Give me a break. Filesystem performance for journalled filesystems is mostly bound by writing speed, and this is a function of how the journal is updated. I doubt you would notice the difference in read speed unless you ran a million tests over a million different files, took some sort of average for the filesystems and quibbled over a few milliseconds.
Reiser4 isn't mature enough to entrust my digital assets to
You entrust your assets digitally? Shit, why do you trust any filesystem? They are all buggy. Give me a break.
If you don't like it, keep backups on other media; buy a tape drive and a robot and get in bed with a good archiving company to securely store the backups. Don't come one here and poo poo all of the file systems known to man then tell me "is there anything better"? About the only 4 in common use you left out were JFS (good for large databases but not much use if you have a lot of small files), FAT[12/16/32] (not much good for anything really), NTFS (see FAT, but more complex) and ISO9660. I'll concede there are others, but if you want something that's in common use so you can actually retreive your data when the world turns to shit...
Anywho!
I drink to make other people interesting!
Comparison of FileSystems (from Wikipedia)
;P
Personally, I run two 300GB drives in RAID1 on UFS and am quite satisfied with it, but you seem to be incredibly, incredibly picky, so I'm sure you could find something wrong with it
ND
This statement is forty-five characters long.
Around 1997, I discovered the magic of mpeg-layer3. I hung out in #mpeg3 on effnet and was part of what was probably the first ever mp3 trading circle. An aquaintance of mine had a CD of the rare Nirvana/Jesus Lizard single, which had Nirvana's "Oh The Guilt" on it. I borrowed it from him and ripped it to wave and encoded it a 256KB mp3 and returned the CD. Over the next year or so, quite a few people nabbed the song from me during normal trading sessions in #mpeg3. Sometime later I made a boo-boo and lost a folder permanently, and one of the files in it was that song. I was bummed, as the person I borrowed the CD from was gone and the CD was long out of print and cost a lot of money if you happened to find a copy. I forgot about it.
Quite a few years later - I think ~2002, I was on some p2p app, typed in "Oh the Guilt" and got a hit. I downloaded it, and it was a 256KB mp3 of the song. The file modification date in 1997, and the tags were typed in exactly the I would have put them if I had encoded the song. I can't prove it, but I'm pretty sure I got my file back.
I don't always use unix-like operating systems; but when I do, I prefer FreeBSD.
While it sucks you've lost data because of XFS, mant people use it heavily every day without issue (I'm one of them) I've deployed XFS across mail, database, and web servers without issue. Your statements about are total FUD. The reason the last 'release' was in 2003 is not long after that, XFS was accepted into the kernel itself. Thus there we no longer a need to 'release' XFS patches for the kernel. If you look at the command packages, you'll see them being updated on a regular basis.
As for bugs, I think your statement of bugs not being fixed is incorrect as well. Check the closed bug list. You'll see many that are being closed. Also, in your open bug list above, it does appear rather long. But MANY of those bugs are from users who opened a bug saying 'XFS Crashed On Me' and then never followed up with more info. The XFS developers haven't cleaned many of those out it seems. Bugs in the 200s date from 2003, bugs from the 300's from 2004. Late 300's and 400's from 2005.
So I hate you've had data loss - I wouldn't wish that on anybody (having experienced a RAID5 triple disk failure combined with backup tape failure. Thank goodness for OnTrack!) But don't post FUD about a filesystem that has performed very well for a lot of people and continues to be improved and innovative.
Top Most Bizarre/Disturbing Error Messages