Best Filesystem For External Back-Up Drives?

rufey writes "I've recently embarked on a project to rip my DVD and CD collection to a pair of external USB drives. One drive will be used on a daily basis to access the rips of music and DVDs, as well as store backups of all of my other data. The second drive will be a copy of the first drive, to be synced up on a monthly basis and kept at a different location. The USB drives that I purchased for this are 1 TB in size and came pre-formatted with FAT32. While I can access this filesystem from all of my Windows and Linux machines, there are some limitations." Read on for the rest, and offer your advice on the best filesystem for this application. "Namely, the file size on a FAT32 filesystem is limited to 4GB (4GB less 1 byte to be technical). I have some files that are well over that size that I want to store, mostly raw DVD video. I'll primarily be using these drives on a Linux-based system, and initially, with a Western Digital Live TV media player. I can access a EXT3 filesystem from both of these, and I'm thinking about reformatting to EXT3. But on Windows, it requires a 3rd party driver to access the EXT3 filesystem. NTFS is an option, but the Linux kernel NTFS drivers (according to the kernel build documentation) only have limited NTFS write support, only being safe to overwrite existing files without changing the file size). The Linux-NTFS project may be able to mitigate my NTFS concerns for Linux, but I haven't had enough experience with it to feel comfortable. At some point I'd like whatever filesystem I use to be accessible to Apple's OS X. With those constraints in mind, which filesystem would be the best to use? I realize that there will always be some compatibility problems with whatever I end up with. But I'd like to minimize these issues by using a filesystem that has the best multi-OS support for both reading and writing, while at the same time supporting large files."

Comments

The solution..

[Anrego]

is to stop being so diverse! Pick a platform and stick with it!

Ok, in all seriousness.. here's what you do:

- buy yourself a cheap (~200) box
- hook all your drives to it
- use whatever file system you want (JFS, XFS would be my recommendation)
- share it over your zoo of a network using nfs, samba, etc..

As a bonus, your file server box could double as a media center, and replace your WD TV Live dealie.. (probably not though.. right)

Irregardless, I think you're way better off with this approach vice trying to find the magical widely supported cross platform file system with large file capacity.

You also might want to consider RAID vs. your monthly sync. Yes, RAID isn't a backup.. but for something like this where
restoration would be possible, but just a pain in the ass.. mirrored raid can be a lot more convenient. You can always have
a third external to back up your irreplacable data on a semi-annual basis..

Re:The solution..

[rpresser]

It might if irregardless was actually a word.

Re:The solution..

[multimediavt]

WRONG!

Irregardless is not a proper English word. Its usage has *ALWAYS* irked me from when I was a small boy to now. To use common vernacular, it's a mashup of 'irrespective' (one negative; prefix) and regardless (also, one negative; suffix). 'Irregardless' is a double negative and is thusly illogical by construction and would only be understandable to people born in the U.S. since 1970, and those less literate in the U.S. prior to that.

On my words that aren't words list it's right up there with 'impactful'.

Re:The solution..

[molarmass192]

You're words are truthy enough, but your assuming that synergistic words like irregardless don't have impacts on english as we know it. The facts is that people will use words like that wether we like it or not. This is truely, the case when it comes to American's use of language. Sadly, theirs very little we, as people far more litterate than the average people, can really do about that. If people used grammer checkers, then you and me would not see so many people authoring bad words and having a negative affect on english as it is known and practised today but should be editted and spokened tomorrow.

Re:The solution..

[EdIII]

I've seen it used multiple times in this discussion. That makes it a word.

ummmm. okay. "Fugnutish". Let's keep it going a few more replies and I gots my new word :)

Re:The solution..

[Tanktalus]

No, most people use it because they think they sound smart when they use a big word. The problem is, it's not a word and thus they just sound like an idiot to the very people they are trying to impress when they say or write irregardless.

As a former coworker once told me, "Never use a large word when a diminutive one will suffice." I think he was showing off.

Re:The solution..

[camperdave]

What's the difference between irregardless and regardless?

Irregardless works in the dark.

Re:The solution..

[yukk]

Utter rubbish. Irregardless is a perfectly cromulent word.

Exactly. It embiggens the language.

Re:Ext3

[moderatorrater]

I personally go the other way. Sure, there's an Ext3 driver for windows, but from what I've seen it's not that good. On the other hand, I've used the NTFS driver on Linux quite a bit and it's worked pretty well. More importantly, I have confidence that the NTFS driver will continue to get better.

The best is...

[FreemanPatrickHenry]

...ReiserFS. I hear it's killer.

Words of caution

[RenHoek]

I have ~6TB on external USB drives and I've been doing this for a few years now. I have a few words of caution about NTFS. If you get an USB drive that for example spins down or if you turn your USB drive off without properly dismounting it (or if Windows crashes), you might see this line:

Delayed write failed!

And on two occasions that meant that Windows fucked up the file allocation table or whatever it's called under NTFS and I lost the _entire_ disk.

Windows loves getting its fingers into that table whenever you mount a USB filesystem. It's not like it tries to keep its write cache empty. Nooo.. every file access needs to be continuously recorded in that thing.

Anyway, be careful when you use NTFS on a USB drive. Alternatively use EXT3, which you can still mount under Windows using:

http://www.ext2fsd.com/

(Note that these experiences are under Windows XP, I have no clue if Vista or 7 does any better, I assume not.)

Re:Ext3

[Sophira]

Sure, there's an Ext3 driver for windows, but from what I've seen it's not that good.

Which particular driver are you referring to? There are a few.

Personally, I use Ext2 IFS in Windows (it works for Ext3 too) and it is, hands-down, the stablest and best Ext2/3 Windows driver I've used. Every other one I've tried would have stability issues; with IFS I don't have to worry. (There's been precisely *one* time in pretty much years that the driver crashed on me, and that's when I was doing something weird and stupid; I don't remember what. But more importantly, it didn't do anything bad to the filesystem in that crash.)

Why all those ext3 recommendations?

[the_other_chewey]

I see quite many people here recommending ext3. Oh my. ext3 sucks for large files,
which is exactly what the submitter wants to use his setup for. Look into the crazy structures
("double indirect blocks") it uses. He should go with an FS that has sane data structures with
files >>4GB.

That kills most of the choices and leaves XFS, ext4, ZFS (only worth it if not used via FUSE,
i.e. in Solaris), and a couple more obscure ones.
I second the "forget OS portability, use a server" suggestion. There's great low-power, low-cost
hardware for this nowadays.

Please read this

[Anonymous Coward]

Dear AP,

I was pondering with a very similiar problem two years ago, when I bought my first terabyte-class external USB hard drive to my home server (old 800 MHz Duron). I was thinking exactly like you are thinking now, and decided FAT-32 would be the way to go. MISTAKE. Four important things stand out, that I want you to read:

1. You never really transfer that drive. Much less than you think. Chances are more than 99% of their lifetime they will be sitting hooked to one computer, never really being moved; portability is just another nice extra feature that you hardly ever use. During the two years I have switched my drives a couple of times between the desktop machine and the server, both of which run Linux as main OS. I have never ever taken any of these drives outside of my home.
2. 4 GB limitation is really bad! Most DVD ISOs are bigger that that. An hour of HD video in state-of-the-art H.264 is more than 4 GB. And rest assured, when you have the space and facilities to accquire gigabyte-class multimedia, the temptation will be there. As BluRay becomes the new DVD, maybe you want to RIP your fav. movies to your hard drive for quick access? NO WAY with FAT-32!
3. Lack of UNIX parameter support. Okay, so you just want to store back-ups? Okay.. Remember than FAT-32 doesn't support symbolic links, file ownership, user/group/others access permissions, file name character case (in Microsoft Windows, "Soviet Union" equals to "soviet union"; WILL result in a conflict when copying data from UNIX filesystems!). This information is LOST, unless you use some container format like tar (but remember the 4GB limit again). These little things are a) very helpful everyday things, value of which you realize only after loosing them (e.g. any file on extfs can be replaced/virtualized without moving files around; it can even point to a non-existstent file! And all works seamlessly, as long as the program understands symbolic links; now how valuable is that?), and b) what makes your UNIX fs work. The value of your backups is lower if they dont work "out of the box", e.g. data is lost when transfering to FAT-32. I mean, you just have a chance to save so much hassle there.. When needed, you can NFS-mount the filesystem (and its free space and contents) to your local machine from your drive, and everything works transparently to BOTH Windows and Linux (the properties of FAT-32 are a small subset of those of extfs.)
4. Acccess speed. Ext3 and Ext4 or just about any 21st century UNIX fs are lightyears ahead of the archaic FAT in data structures. E.g. if I "ls" a big directory on my only FAT-32 drive, it is SLOW! You can see the entries being fetched one by one. Whereas, if I do the same in a similiarly-sized directory on the ext3, the files appear immediately! Access is almous immediate even over NFS mount in LAN. This comes handy, rest assured.

Okay, those are my four vocal points. They could be in any order, because all of them are equally important reasons NOT to choose FAT-32! As it happened, after using the 0.5TB drive for 6 months with FAT-32, I bought more space (a new drive). This time there was no question about the filesystem. I made a small, few-sector long 200 MB FAT-32 partition to the beginning of the drive and downloaded all the latest Win32 EXT2 drivers there from different vendors, just for the really unplausible situation that I would ever want to mount these drives in Windows. Then I just made the rest Ext3. And, I am REALLY satisfied with the decision! Ext3 just work so sparklingly faster and better with Linux than FAT-32 ever does. Since then I have bought one more drive and did the same 200MB + 1TB thing. I will probably never use these drives in Windows, but it gives me a warm feeling to the heart that there's always a way if I should, even if the computer doesn't have an Internet connection.

Oh, one thing I forgot to mention: get a file server! It makes your life so much easier. Nowadays I am running a desktop computer with a 60 GB SSD drive and no HDD at a

Problem with ReiserFS

[TheLink]

The big problem with ReiserFS: Vendor lock-in.