A Good Filesystem for Storing Large Binaries?

jZnat asks: "I own hundreds of gigabytes of binary data, usually backed up from other mediums such as CDs and DVDs. However, I cannot figure out which filesystem would be best for storing all this reliably. What I'm looking for is a WORM-optimized FS that also has good journaling methods to prevent data loss due to some natural disaster while data is being shifted around. Trying something new for once, I tried using SGI's XFS due to its promising details, but I was met with countless IO errors after trying to write large amounts of data to it. I feel that Ext3 is not optimal for this; ReiserFS is too slow when it comes to reading large data files; and Reiser4 isn't mature enough to entrust my digital assets to. What filesystem would be most appropriate for these needs?"

Comments

The Google Filesystem

[benploni]

Google made a filesystem for exactly that purpose: storing HUGE files highly reliably. OK, so it's not publically available, but it's still perfect for you (other than that).

Possibly...

[dcapel]

p2pfs?

Just upload to bittorrent, ftp, or some other p2p system, and redownload it if you need it again!

Some small security issues may apply though...

TAR files written to raw partitions

[vrmlguy]

Don't laugh! Most (if not all) filesystems are optimized to handle the opposite of what you want. TAR files are designed for tape, so you won't be seeking all over the disk to get meta information, instead you'll get your data at the maximum speed supported by your hardware. TAR files are designed so that you can append files to them later, so you can use a *big* partition and just keep dumping stuff into it.

The only drawbacks are that you have to read the entire partitioin sequentially to find things, and you can't delete files. Both of these can be fixed with a bit of Perl. Write a program that maintains an index of offsets to the files, then you can use "dd" to skip to the correct offset and read from there. More dangerously, write a program that deletes files from the middle of an archive and shuffles everything backwards to fill in the gaps. You'll want to make sure that no one is trying to read the TAR partition while this is running.