A Good Filesystem for Storing Large Binaries?

jZnat asks: "I own hundreds of gigabytes of binary data, usually backed up from other mediums such as CDs and DVDs. However, I cannot figure out which filesystem would be best for storing all this reliably. What I'm looking for is a WORM-optimized FS that also has good journaling methods to prevent data loss due to some natural disaster while data is being shifted around. Trying something new for once, I tried using SGI's XFS due to its promising details, but I was met with countless IO errors after trying to write large amounts of data to it. I feel that Ext3 is not optimal for this; ReiserFS is too slow when it comes to reading large data files; and Reiser4 isn't mature enough to entrust my digital assets to. What filesystem would be most appropriate for these needs?"

Comments

JFS

[member57]

I use JFS on RAID 5, no errors, uptime of 200+ days currently. Handling large files 200-300MB each all day long. Excellent performance.

Comparison of File Systems

[NuclearDog]

Comparison of FileSystems (from Wikipedia)

Personally, I run two 300GB drives in RAID1 on UFS and am quite satisfied with it, but you seem to be incredibly, incredibly picky, so I'm sure you could find something wrong with it ;P

ND

I/O Errors???

[Stephen Samuel]

If you're getting lots of I/O errors with XFS, I'd be inclined to look at a hardware problem (unless the I/O errors consist of attempts to read past the end of the partition -- which could be caused by you manually specifying the partition size, rather than letting mkfs.xfs figure it out).

Like someone else said -- try using badblocks(8) -- or just use dd to make sure you can read the entire partition without errors.
Bad disks do happen -- even new ones. Production code in Linux is generally very stable, and (unlike with windows), you can usually start with the presumption that things like I/O errors are caused by real hardware problems of some sort (even if it's just bad/loose cables).

Re: ext3 works fine, did you try it?

[Matt Perry]

I feel that Ext3 is not optimal for this

Did you try it with ext3? I have 688G in a RAID5 array spread across four 250GB drives. I use ext3 and I store lots of large files (15GB free on the array right now). I have about 156GB of DVD images, mostly movies that I own and have ripped to watch using daemon tools on Windows. Some of them are rips of training video DVDs I bought for software that I use like Adobe Premiere and Audition. I frequently move large AVI files to and from the array for video projects that I'm working on. These files originate on my Windows box and can be as large as 13GB (for an hour of video footage). I've been using ext3 for years and it's never let me down or given me any problems.

Re:Ext3 or XFS.

[baptiste]

Check out the latest. What? 2003? Haven't there been any bug fixes since then?

While it sucks you've lost data because of XFS, mant people use it heavily every day without issue (I'm one of them) I've deployed XFS across mail, database, and web servers without issue. Your statements about are total FUD. The reason the last 'release' was in 2003 is not long after that, XFS was accepted into the kernel itself. Thus there we no longer a need to 'release' XFS patches for the kernel. If you look at the command packages, you'll see them being updated on a regular basis.

As for bugs, I think your statement of bugs not being fixed is incorrect as well. Check the closed bug list. You'll see many that are being closed. Also, in your open bug list above, it does appear rather long. But MANY of those bugs are from users who opened a bug saying 'XFS Crashed On Me' and then never followed up with more info. The XFS developers haven't cleaned many of those out it seems. Bugs in the 200s date from 2003, bugs from the 300's from 2004. Late 300's and 400's from 2005.

So I hate you've had data loss - I wouldn't wish that on anybody (having experienced a RAID5 triple disk failure combined with backup tape failure. Thank goodness for OnTrack!) But don't post FUD about a filesystem that has performed very well for a lot of people and continues to be improved and innovative.