Slashdot Mirror
A Good Filesystem for Storing Large Binaries?
jZnat asks: "I own hundreds of gigabytes of binary data, usually backed up from other mediums such as CDs and DVDs. However, I cannot figure out which filesystem would be best for storing all this reliably. What I'm looking for is a WORM-optimized FS that also has good journaling methods to prevent data loss due to some natural disaster while data is being shifted around. Trying something new for once, I tried using SGI's XFS due to its promising details, but I was met with countless IO errors after trying to write large amounts of data to it. I feel that Ext3 is not optimal for this; ReiserFS is too slow when it comes to reading large data files; and Reiser4 isn't mature enough to entrust my digital assets to. What filesystem would be most appropriate for these needs?"
I use JFS on RAID 5, no errors, uptime of 200+ days currently. Handling large files 200-300MB each all day long. Excellent performance.
If Kerry was the answer, it must have been a stupid question.
The UN - The largest "political" cause of death.
Google made a filesystem for exactly that purpose: storing HUGE files highly reliably. OK, so it's not publically available, but it's still perfect for you (other than that).
I've been using JFS for about two months now, and it's been quite a plesant experiance with my anime storage. I run it on a 1.6TB array, four 400GB harddrives. It's preformance is damn fast from what I've observed copying too/from a JFS firewire drive. I trust it enough to keep data that I can't back up on it until I can get another identicle array to mirror with - only drive failure will seem to kill this FS, and these drives are about ~3 months old, so failure isn't that much of a concern anymore. I'd reccomend it for storage, havn't tried it as a system FS yet.
So lets get this straight:
You need a filesystem that can be "burned" to a medium, yet have error correction capability.
Journaling doesn't do this. Journaling is for when you get a power surge in the middle of a write, you can get some of the data back. Currently no regular FS can do that.
--
# Canmephians for a better Linux Kernel
$Stalag99{"URL"}="http://stalag99.net";
I know this is not exactly, what you are looking for, but database companies have very similar problem on their hands and since filesystems usually are not quite good for this type of work, they usually come up with their own systems for handling raw disks. For example Oracle has its ASM (Automated Storage Management). You might want to look into these if they are not customizable for your problem or contact the relevant companies for specifics.
If programs would be read like poetry, most programmers would be Vogons.
Okay, why is this modded troll again? He misunderstood the direction of the backups, he wasn't trolling! If anything, it's -1 WRONG, not troll! Anyway - he backs up data FROM DVDs and CDs, not TO
reiser or jfs are both solid for this kind of work, with large file and volume support. personally i swear by reiser for my 2tb volume, and have had no problems so far, although there is a minor speed penalty when working with several multi-gigabyte files at once, something to do with shared fs locks/mutexes i'd imagine.
OTOH JFS is quite stable, and though it has less of the elegance in feature set I find in reiser, tends to make up for it with enhanced ruggedness and its handling of large volume/files.
Really can't recommend anything else, as you say, reiser4 is still untested for reliability imho, xfs has issues that vary from kernel to kernel, and ext3 appears quite primitive in comparision, although its journaling seems comparable to the other choices.
JFS if you need the speed, its dead fast in large scales, slower with small files, otherwise Reiser3 is an excellent all-round performer.
The first rule of USENET is you do not talk about USENET.
p2pfs?
Just upload to bittorrent, ftp, or some other p2p system, and redownload it if you need it again!
Some small security issues may apply though...
DYWYPI?
ZFS has some built-in volume management & data integrity functions that would probably work for you. I don't believe that it is available for Linux, but is freely available via Solaris & OpenSolaris
http://www.sun.com/software/solaris/zfs.jsp
Conformity is the jailer of freedom and enemy of growth. -JFK
Most fancy filesystems like ReiserFS are optimized for performance with lots and lots of tiny files where the disk reads little at a time, seeking, sorting, assembling, slicing etc take most of the time. Here you have few big files, so performance is your least worry - the harddrive read/write speed will be the bottleneck, and all the seeks, directory reads etc will be scarce and fast. Therefore the filesystem won't change much in the means of speed. (it MAY break a lot in the department, like, say compressed filesystems, but won't speed it up above what the harddisk does, and most of filesystems will perform just the same in the means of speed.) What you can do is to optimize the filesystem for capacity, reducing its overhead and allowing to get closer to "advertised disk capacity".
Just use tune[23]fs to reduce number of inodes significantly on the ext3fs. Or look for -simple- filesystems that don't do tricks in optimization of speed (because these usually waste diskspace), just store your files in a straightforward manner.
Anagram("United States of America") == "Dine out, taste a Mac, fries"
journaling methods to prevent data loss due to some natural disaster while data is being shifted around
Journalling doesn't do this!. Journalling helps reduce file system corruption in the event of a catastrophic failure while modifying the file system - ie, it's possible to bring it back to the last clean state before it crashed - journalling does not prevent data loss. You might say "well filesystem corruption and data loss are the same", but they are not. If the filesystem is corrupted, the data is not lost. It just becomes not easily retreivable. If the data is lost then it becomes entirely irretreivable.
I tried using SGI's XFS due to its promising details, but I was met with countless IO errors
Have you considered your hardware is shit? I use XFS on terabytes of raided disks and have been for more years than I remember... 5 or so? I don't see any I/O errors. XFS is very reliable and I trust it with my data.
I feel that Ext3 is not optimal for this
Well not all of your post was dumb!
ReiserFS is too slow when it comes to reading large data files
How is it slow? It takes a few microseconds longer to access the first data sector because it does some extra processing first? Give me a break. Filesystem performance for journalled filesystems is mostly bound by writing speed, and this is a function of how the journal is updated. I doubt you would notice the difference in read speed unless you ran a million tests over a million different files, took some sort of average for the filesystems and quibbled over a few milliseconds.
Reiser4 isn't mature enough to entrust my digital assets to
You entrust your assets digitally? Shit, why do you trust any filesystem? They are all buggy. Give me a break.
If you don't like it, keep backups on other media; buy a tape drive and a robot and get in bed with a good archiving company to securely store the backups. Don't come one here and poo poo all of the file systems known to man then tell me "is there anything better"? About the only 4 in common use you left out were JFS (good for large databases but not much use if you have a lot of small files), FAT[12/16/32] (not much good for anything really), NTFS (see FAT, but more complex) and ISO9660. I'll concede there are others, but if you want something that's in common use so you can actually retreive your data when the world turns to shit...
Anywho!
I drink to make other people interesting!
I have a 3 TB XFS file system and a 10 TB XFS file system that are regularly accessed by multiple processes that read and write hundreds of gigabytes each without write errors and with excellent performance (several hundred MByte/s sustained). You may have other hardware or software issues if you're seeing errors with XFS. Try to figure out the root cause of your problem before you try another FS.
As a general rule, the latest and greatest stuff will be full of bugs. Give zfs a few years before you trust it with anything critical.
Comparison of FileSystems (from Wikipedia)
;P
Personally, I run two 300GB drives in RAID1 on UFS and am quite satisfied with it, but you seem to be incredibly, incredibly picky, so I'm sure you could find something wrong with it
ND
This statement is forty-five characters long.
Like someone else said -- try using badblocks(8) -- or just use dd to make sure you can read the entire partition without errors.
Bad disks do happen -- even new ones. Production code in Linux is generally very stable, and (unlike with windows), you can usually start with the presumption that things like I/O errors are caused by real hardware problems of some sort (even if it's just bad/loose cables).
Free Software: Like love, it grows best when given away.
If you're looking for a filesystem to archive things indefinitely, avoid exotic new kids on the block with limited OS support and even more limited toolkit support.
You want a filesystem you'll be able to read at any point in the future and, should the worst happen, one which you'll have a reasonable chance of being able to recover.
ext2 and fat32 tend to write files in nice large chunks and there are lots and lots of recovery tools for damaged filesystems. Journaled filesystems like to put little pieces all over the place, and recovery of a badly damaged filesystem is next to hopeless.
There is no call for a complex filesystem just because you want to store large files. ext2 (and to some extent fat32) will do just fine, and you'll be glad for them someday in the future when something breaks.
Around 1997, I discovered the magic of mpeg-layer3. I hung out in #mpeg3 on effnet and was part of what was probably the first ever mp3 trading circle. An aquaintance of mine had a CD of the rare Nirvana/Jesus Lizard single, which had Nirvana's "Oh The Guilt" on it. I borrowed it from him and ripped it to wave and encoded it a 256KB mp3 and returned the CD. Over the next year or so, quite a few people nabbed the song from me during normal trading sessions in #mpeg3. Sometime later I made a boo-boo and lost a folder permanently, and one of the files in it was that song. I was bummed, as the person I borrowed the CD from was gone and the CD was long out of print and cost a lot of money if you happened to find a copy. I forgot about it.
Quite a few years later - I think ~2002, I was on some p2p app, typed in "Oh the Guilt" and got a hit. I downloaded it, and it was a 256KB mp3 of the song. The file modification date in 1997, and the tags were typed in exactly the I would have put them if I had encoded the song. I can't prove it, but I'm pretty sure I got my file back.
I don't always use unix-like operating systems; but when I do, I prefer FreeBSD.
Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
Beyond that, I'd say pretty much anything will work fine -- most of the optimizations found in filesystems are needed for lots of small files, not a few large files. For large files, the speeds they can be accessed by various filesystems are not likely to vary more than a few percent unless you let the files get fragmented (which probably isn't a big concern here.)
And you are right -- if something does go wrong, ext2 or ext3 will probably give you the most options for recovering it. NTFS probably has even more recovery options (and FAT even more, as mentioned), but I'm guessing the OS will be *nix. But really, if your goal is reliability, you don't want some esoteric filesystem that can recover from disk errors (because ultimately, none can, though I guess one could be designed to keep ECC codes on the same disk transparantly -- but I'm aware of no such filesystem existing) -- you want multiple copies of your data. Keeping 5-10% (or more) par2 files for your archive can help a lot in recovering it if your media goes partially bad, and having md5sums or CRC32s of all archived files can help determine if you did recover something accurately, but really there's little subsitute for multiple copies of important data in multiple geographical locations. (And no -- RAID is not a subsitute for backups, no matter how many mirrored drives you have. Not that I saw anybody suggest this yet, but it seems to always come up in response to questions like this, so consider this to be a premptive mention of that.)
Ext2fs mounted with the 'sync' option.
For large sequential writes, nothing could possibly be more reliable or any faster. Your hard drive's pure IO speed will be the bottleneck unless you are writing to multiple files simultaneously, in which case fancy filesystems come in handy.
If that doesn't suit your needs, you haven't described them well enough for anyone to understand.
I feel hungry.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
Nothing, the programmers will tell you that themselves. Journalled filesystems aren't for protecting your files. They're for protecting your filesystems.
The point of a journal is that you can roll back to a consistent state of the filesystem easily in case of error -- not that you can roll back to a consistent state for a given file (or indeed any file on the filesystem). In point of fact, it's usually more difficult to recover actual data from a journalled filesystem than a traditional one, because the writing process is much more complex. What's more, if an automatic fsck is needed, it's actually a little more likely to lose some data on a journalled filesystem because a non-journal filesystem recovers based on the found files while a journalled one recovers based on a separately recorded journal (you do put your journals on a separate block device from the filesystem, right? Otherwise you're mostly wasting your effort.)
The main point of a journalled filesystem is that when both redundant circuits in your datacenter go at once (which happens) the boxes will be able to at least boot and get through the automatic fscks without a tech needing to drive out there and run the fscks himself.
All's true that is mistrusted
You must be really fun at parties.
For all my WORM disks, I use either ISO 9660 or ISO 9660/UDF bridge format.
Yeah, I simply burn CD-Rs or DVDs. DVDs have the nice property of being easily stored off-site. And files are in nice large contiguous block so even if the filesystem dies you can still recover a lot. Unlike XJFReiFS 2.3.1.5, DVDs will be readable in 50 years time.
And if you need to burn really large files, just use, well, zip. And perhaps some par2 files.
Though, seriously, they're coming up with a UDF variant for hard drives too.
SCO employee? Check out the bounty
While it sucks you've lost data because of XFS, mant people use it heavily every day without issue (I'm one of them) I've deployed XFS across mail, database, and web servers without issue. Your statements about are total FUD. The reason the last 'release' was in 2003 is not long after that, XFS was accepted into the kernel itself. Thus there we no longer a need to 'release' XFS patches for the kernel. If you look at the command packages, you'll see them being updated on a regular basis.
As for bugs, I think your statement of bugs not being fixed is incorrect as well. Check the closed bug list. You'll see many that are being closed. Also, in your open bug list above, it does appear rather long. But MANY of those bugs are from users who opened a bug saying 'XFS Crashed On Me' and then never followed up with more info. The XFS developers haven't cleaned many of those out it seems. Bugs in the 200s date from 2003, bugs from the 300's from 2004. Late 300's and 400's from 2005.
So I hate you've had data loss - I wouldn't wish that on anybody (having experienced a RAID5 triple disk failure combined with backup tape failure. Thank goodness for OnTrack!) But don't post FUD about a filesystem that has performed very well for a lot of people and continues to be improved and innovative.
Top Most Bizarre/Disturbing Error Messages
The only drawbacks are that you have to read the entire partitioin sequentially to find things, and you can't delete files. Both of these can be fixed with a bit of Perl. Write a program that maintains an index of offsets to the files, then you can use "dd" to skip to the correct offset and read from there. More dangerously, write a program that deletes files from the middle of an archive and shuffles everything backwards to fill in the gaps. You'll want to make sure that no one is trying to read the TAR partition while this is running.
Nothing for 6-digit uids?
The number of inodes in a filesystem is an option that can only be specified upon filesystem creation. i.e. to mke2fs, not tune2fs.
retrorocket.o not found, launch anyway?
What you're looking for is Universal Disk Format or UDF.
It is an open standard supported by all of the major OSes and manufacturers and is the filesystem of choise for Ultra Density Optical WORM and rewritable disks.
There a drivers for Linux, Windows and all of the major UNIXes. Here is the obligatory Wikipedia entry.
Hard disk filesystems like XFS, JFS, Reiser, ZFS etc. are all wonderful at what they do but they are unsuitable for WORM disks.
Stick Men
Others have said good things in general (XFS,JFS,ext3).
I looked into filesystem comparisons in setting up a MythTV box.
My issues were:
(1) efficient use of hard drive space, and
(2) performance.
Efficient use = filesystem settings have a big effect on amount of usable space.
For ext2/3:
-m 0 = setting 'reserved space for root' to 0%. Default is 5%, which can be 10-20 GB these days, all unusable to non-root users
-T ____ = can tell ext2/3 to optimize inodes and byte-per-inode for different size average files. Largefile versus news spools (tons of small files). Because of the way that a file can be spread out and mapped across the filesystem, this has an effect on 'wasted' space, and maybe performance (# of inode entries per file to lookup).
-b, -i - can set total # of inodes and bytes-per-inode directly. Advanced control over filesystem creation
I never got around to looking into this detail for XFS/JFS - they seem have fewer such options.
Performance I'll leave it to others to talk about filesystem performance with largefiles in general.
MythTV takes a lot of writing, and as it turns out, deleting, of large temporary files for the TV features (records, pause, FF/RR). After some reading online, I've found MythTV performance is drastically impacted by filesystem choice due to all of the deleting.
http://www.mythtv.org/docs/mythtv-HOWTO-24.html#ss 24.2
http://www.gossamer-threads.com/lists/mythtv/users /52672
---SNIP---
> My last reply to myself. Based on a Googled reference, I was able to
> break my XFS 4G file size barrier by formatting the partition 'mkfs.xfs
> -dagsize=4g'. So, here are the complete results:
>
> Time to delete a 10G file, fastest to slowest:
>
> JFS: 0.9s, 0.9s
> XFS: 1.3s
> EXT3: 1.4s, 2.3s
> EXT2: 1.6s
> REISERFS: 6.2s
> EXT3 -T largefile4: 5.9s, 10.2s
>
> After running the XFS test, there didn't seem to be any point in
> reformatting the partition again, so I left it on XFS, but I think I
> would be happy with JFS, XFS, or EXT3 w/o '-T largefile4'.
>>>>
wepprop at sbcglobal
Feb 8, 2004, 2:33 AM
Post #21 of 22 (4121 views)
Re: Changing filesystems? [In reply to]
Robert Kulagowski wrote:
> Interesting. If others care to weigh in, I can either re-write the
> "Advanced Partitioning" section in the HOWTO, or whack it completely.
>
> William, can you give some background on the hardware used for your
> tests? I'd be curious if this data holds up across various drive types,
> LVM, etc. (Without trying to exhaustively test all the possibilities,
> that is)
It appears, based on my personal experience alone, that file deletes are
the only system operations that can stress the hard drive enough to
produce dropped frames. Unfortunately, as others have pointed out,
recordings and deletions go together in Myth. So, unusual as it may be,
it does make at least some sense to take file deletion performance into
account when deciding which filesystem to use for a video partition,
especially for people with multiple tuners.
The really ironic result from my personal perspective is that it would
appear that using the '-T largefile4' setting for ext3, which I was so
pleased with because it give me an extra 2G of storage, may well have
been responsible for all those recordings I had ruined by frame drops.
Assuming it works out, though, I could really get to like this XFS
filesystem because it appears to give me slightly more storage space
than ext3 w/ '-T largefile4' did and it has pretty fast deletes as well.
---SNIP---
You're obviously looking for a filesystem optimized for porn. I'm impressed that you've managed to accumulate hundreds of gigs of the stuff. Perhaps there is a Porn File System out there somewhere?
Honestly, since I submitted this about a couple months ago, I just formatted the disks to ext3 and it's worked quite well since then, but any better ideas are always welcome.
'Yes, firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'
> usually backed up from other mediums such as CDs and DVDs
This is a very euphemistic way of saying:
"I download moviez, mp3 and porn via P2P all day and even though I usually don't view any movie twice, I still don't want to throw away anything, because I just can't delete anything".
How that could get an "ask slashdot"-posting is left as an exercise to the reader.
Windows 2000 - from the guys who brought us edlin
But, after trying just about every FS under the sun for my backups, on Linux, FreeBSD, and OS X, I finally settled on Mac HFS+ with journaling and case-sensitivity enabled. I have a 900GB RAID with it on it, and I'm storing some files that are 7GB+. I haven't had any issues with it at all.
Yep, it means you will probably need a mac, but Linux does have HFS support (I don't know how good it is). But everything is working out great, and supposedly has some sort of auto-defrag, but I'm too lazy to actually verify this.
Need Free Juniper/NetScreen Support? JuniperForum
Which is why I asked if the poster was serious about using WORM or not.
ISO-9660 is not the same as UDF. If you have UDF and ISO-9660 on the same volume it is because some one mastered a hybrid filesystem structure onto the disc. Which was the norm on first generation DVD's.
ISO-9660 contains no optimizations for being a WORM filesystem, there are no linking records in ISO-9660 to allow re-writing of data into new blank spots on the non-rewriteable storage media, UDF supports these linking blocks.
When I wrote a UDF filesystem for Linux I tested it by building the structures into blocks on hard disk storage devices. It can be done but UDF isn't designed as a high performance FS, but rather as a highly interchangeable FS.
It's a commercial product from Siemens, which I used years ago for Sietec's large-scale imaging product.
There is probably a Linux port: We ran it on almost everything in existance (;-))
--dave
davecb@spamcop.net
I never run a computer without a UPS. IMO, they should be built into the power supplies (I wonder if you can buy such things...). I guess I've had too many computers die on me due to power issues, but power can be very unreliable in my location. UPSes are so cheap it's not worth it to *not* use them.
Max.
[completely offtopic]
I always caught shit from other mp3'ers back in the late 90's because of my 'huge' 256kb songs. People that would download from me would frequently complain that my files were too big and that there was no use encoding them at bitrates that high because "128kbps was already CD Quality".
It was also really easy to start flamewars by bringing up the topic. You could just go into an mp3 IRC channel, make an offhand comment like "128kbps mp3 files sound like crap; 192-156 is really needed to approach true CD Quality", and people would immediately start arguing with you - probably in a subconscious effort to justify the fact that they had spent the last three months encoding their entire CD collection at 128kbps.
I don't always use unix-like operating systems; but when I do, I prefer FreeBSD.