Slashdot Mirror
Best Filesystem For External Back-Up Drives?
rufey writes "I've recently embarked on a project to rip my DVD and CD collection to a pair of external USB drives. One drive will be used on a daily basis to access the rips of music and DVDs, as well as store backups of all of my other data. The second drive will be a copy of the first drive, to be synced up on a monthly basis and kept at a different location. The USB drives that I purchased for this are 1 TB in size and came pre-formatted with FAT32. While I can access this filesystem from all of my Windows and Linux machines, there are some limitations." Read on for the rest, and offer your advice on the best filesystem for this application.
"Namely, the file size on a FAT32 filesystem is limited to 4GB (4GB less 1 byte to be technical). I have some files that are well over that size that I want to store, mostly raw DVD video. I'll primarily be using these drives on a Linux-based system, and initially, with a Western Digital Live TV media player. I can access a EXT3 filesystem from both of these, and I'm thinking about reformatting to EXT3. But on Windows, it requires a 3rd party driver to access the EXT3 filesystem. NTFS is an option, but the Linux kernel NTFS drivers (according to the kernel build documentation) only have limited NTFS write support, only being safe to overwrite existing files without changing the file size). The Linux-NTFS project may be able to mitigate my NTFS concerns for Linux, but I haven't had enough experience with it to feel comfortable. At some point I'd like whatever filesystem I use to be accessible to Apple's OS X. With those constraints in mind, which filesystem would be the best to use? I realize that there will always be some compatibility problems with whatever I end up with. But I'd like to minimize these issues by using a filesystem that has the best multi-OS support for both reading and writing, while at the same time supporting large files."
is to stop being so diverse! Pick a platform and stick with it!
Ok, in all seriousness.. here's what you do:
- buy yourself a cheap (~200) box
- hook all your drives to it
- use whatever file system you want (JFS, XFS would be my recommendation)
- share it over your zoo of a network using nfs, samba, etc..
As a bonus, your file server box could double as a media center, and replace your WD TV Live dealie.. (probably not though.. right)
Irregardless, I think you're way better off with this approach vice trying to find the magical widely supported cross platform file system with large file capacity.
You also might want to consider RAID vs. your monthly sync. Yes, RAID isn't a backup.. but for something like this where
restoration would be possible, but just a pain in the ass.. mirrored raid can be a lot more convenient. You can always have
a third external to back up your irreplacable data on a semi-annual basis..
THat's what I use here. for MAC, there's NTFS-3G (free) or Paragon ($ but faster on writes).
I use NTFS on both my machines (Win/OS X/Linux) without any problems.
NTFS-3G is also available for Linux.
I've got better things to do tonight than die.
I personally go the other way. Sure, there's an Ext3 driver for windows, but from what I've seen it's not that good. On the other hand, I've used the NTFS driver on Linux quite a bit and it's worked pretty well. More importantly, I have confidence that the NTFS driver will continue to get better.
VLC can play rar-compressed-splitted files beautifully, So 4GB is not a very big problem
...ReiserFS. I hear it's killer.
I have discovered a truly marvelous
I wouldn't limit myself to a certain filesystem, I'd run a dedicated NAS like FreeNAS and share it over the network via SMB (windows), AFP (apple) and whatever for Linux - all set. Plus as mentioned above, you can run Firefly media server, a bittorrent server, a DAAP server (itunes sharing), etc (all included in FreeNAS. http://freenas.org/) on the same box. And since filesystems don't matter in this config, you can use ZFS to make a RAIDZ pool of your drives. It's what I do now.
fak3r.com
Via FUSE you'll get consistent features and useability across all 3 OSes. Of course moving zfs drives between those OSes isn't something I've tried, but in theory it should work fine.
Not what your asking for, but Id put a FBSD samba server up with ZFS drives. You can still mount them on other OSes later if need be via FUSE.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
Honestly, if FAT32 won't do what you need, NTFS is pretty much where you'll need to go. NTFS-3g gives you stable read/write capability on Linux and OS X as a FUSE driver; in fact, many distributions have NTFS-3g in their repositories. There's also native NTFS write support in Snow Leopard if you want to risk turning it on. I personally haven't had any issues with it, but some people have encountered file corruption when using it, so you might want to be wary. It is worth noting, however, that many embedded devices won't read anything other than FAT. If you plan on hooking this drive up to, say, a DVD player to show pictures, NTFS won't work for you.
Like it or not, Microsoft file systems are the lingua franca of file transfer on portable drives these days, merely due to the installed base of Windows computers.
The Freelance Wizard
I have ~6TB on external USB drives and I've been doing this for a few years now. I have a few words of caution about NTFS. If you get an USB drive that for example spins down or if you turn your USB drive off without properly dismounting it (or if Windows crashes), you might see this line:
Delayed write failed!
And on two occasions that meant that Windows fucked up the file allocation table or whatever it's called under NTFS and I lost the _entire_ disk.
Windows loves getting its fingers into that table whenever you mount a USB filesystem. It's not like it tries to keep its write cache empty. Nooo.. every file access needs to be continuously recorded in that thing.
Anyway, be careful when you use NTFS on a USB drive. Alternatively use EXT3, which you can still mount under Windows using:
http://www.ext2fsd.com/
(Note that these experiences are under Windows XP, I have no clue if Vista or 7 does any better, I assume not.)
Not if you use a network filesystem, such as Samba and NFS for the Windows and MacOS machines. Then on the Linux fileserver side, use whatever filesystem you want, and any OS can talk to that server.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
As an alternative to an external disk that goes to multiple machines, this might cost some, but perhaps consider a backup server?
The advantages to this setup:
1: The server initiates the backups, and can warn you in case something can't be read.
2: Most backup software stores snapshots, and some deal with the full/incremental/different cycle by using synthetic full backups. This makes restores to a certain point in time pretty easy.
3: More sophisticated backup software allows you to transfer backup sets to another media. This way, you just plug in a drive, do a transfer, and you have an offsite archive.
4: If one of the backup client machines gets hacked or malware installed, existing data stored on backup media cannot be altered.
The disadvantages:
1: You will need an active computer which is significantly more expensive than a hard disk.
2: Amanda/Zmanda for open source, Retrospect, Backup Exec, for commercial. The software costs a hefty chunk of change.
3: You have to make extremely sure that the backup server box is locked down tight. If someone compromises your backup server, they got data of every box you have. If you can, perhaps consider buying a router to put the backup server behind and only allowing the vital ports incoming.
4: Backup servers should have some redundancy for stored data. Because there is so much data stored from multiple boxes, a failure of a drive hurts more than on a normal machine.
5: Restoring a machine may vary in difficulty.
If you're going to build a box, why not use FreeNAS with ZFS? It installed to a USB stick and everything is configurable from a web interface.
Which particular driver are you referring to? There are a few.
Personally, I use Ext2 IFS in Windows (it works for Ext3 too) and it is, hands-down, the stablest and best Ext2/3 Windows driver I've used. Every other one I've tried would have stability issues; with IFS I don't have to worry. (There's been precisely *one* time in pretty much years that the driver crashed on me, and that's when I was doing something weird and stupid; I don't remember what. But more importantly, it didn't do anything bad to the filesystem in that crash.)
I haven't tried it but it looks like a good idea. http://www.cyberguys.com/product-details/?productid=36218&sk=MC71419
Format it ext3 and then share it SMB for any OS.
You are lucky, Ed Gruberman. Few novices experience so much of Ti Kwan Leep so soon.
I see quite many people here recommending ext3. Oh my. ext3 sucks for large files,
which is exactly what the submitter wants to use his setup for. Look into the crazy structures
("double indirect blocks") it uses. He should go with an FS that has sane data structures with
files >>4GB.
That kills most of the choices and leaves XFS, ext4, ZFS (only worth it if not used via FUSE,
i.e. in Solaris), and a couple more obscure ones.
I second the "forget OS portability, use a server" suggestion. There's great low-power, low-cost
hardware for this nowadays.
NTFS is recommended several times in response to this article and it troubles me. The next de facto portable filesystem standard is being determined by us in the same way mp3 was made a standard. Ask yourself if you really want to see NTFS become the standard? Personally I'd rather see a truly open filesystem become standard and I really don't want to see Microsoft have the leverage to make more patent threats. If you agree with me and have some integrity, please recommend an open standard.
Dear AP,
I was pondering with a very similiar problem two years ago, when I bought my first terabyte-class external USB hard drive to my home server (old 800 MHz Duron). I was thinking exactly like you are thinking now, and decided FAT-32 would be the way to go. MISTAKE. Four important things stand out, that I want you to read:
Okay, those are my four vocal points. They could be in any order, because all of them are equally important reasons NOT to choose FAT-32! As it happened, after using the 0.5TB drive for 6 months with FAT-32, I bought more space (a new drive). This time there was no question about the filesystem. I made a small, few-sector long 200 MB FAT-32 partition to the beginning of the drive and downloaded all the latest Win32 EXT2 drivers there from different vendors, just for the really unplausible situation that I would ever want to mount these drives in Windows. Then I just made the rest Ext3. And, I am REALLY satisfied with the decision! Ext3 just work so sparklingly faster and better with Linux than FAT-32 ever does. Since then I have bought one more drive and did the same 200MB + 1TB thing. I will probably never use these drives in Windows, but it gives me a warm feeling to the heart that there's always a way if I should, even if the computer doesn't have an Internet connection.
Oh, one thing I forgot to mention: get a file server! It makes your life so much easier. Nowadays I am running a desktop computer with a 60 GB SSD drive and no HDD at a
The big problem with ReiserFS: Vendor lock-in.
Except that it's basically unusable now.
I don't know the details 100%, but the Ext2 IFS driver doesn't support Inodes that are larger than 128 bytes, according to the site you linked.
My understanding is that Ext3 defaults to 256 byte Inodes now. The only way to change it is to manually create the file system with some flags, no gui option, certainly not while I'm installing Fedora or Ubuntu.
I tried getting it to go but when I attempt to access the drives windows tells me it needs to be formatted before use. Not good!
Perhaps it will eventually be fixed, but I will agree with the GP, using NTFS from Linux has been rock solid for at least a couple years for me.
I would recommend NTFS.
That is the best solution to read from, but he says that he's going to primarily be using this from Linux - so NTFS might still be a bit risky.
I would also recommend that you get an open source program and compress the dvd's to approx. 700MB for up to 90 minutes and 1.4GB over 90 mins. H264/ac3 or Xvid are good codecs to use.
A couple of points here. First, why would he specify a file size unless he was going to put the movies on CDs? He'd be far better off simply using a constant-quality encoder, which would have the additional benefit of not requiring a second or third pass for optimal quality. Second, why recompress in this day and age? A 1TB drive can hold over 100 double-layer DVD movies... and if he has more DVDs than that, then the cost of a bigger/additional drive is minor compared to the cost of the collection...
mp3 files are like .jpg image files such that they lose a little quality/data each time they are copied
This is simply not true. Neither mp3s nor jpegs lose quality or data when copied. They only lose quality when they are re-encoded - just as you are proposing he do with his DVDs. DVDs are heavily compressed using MPEG2. Converting to MPEG4 or H264 is exactly the same as taking an MP3 and converting it to AAC or OGG - there will be quality loss no matter how well the encode is done.
W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
The fundamental problem lies in USB bridge chips which do not properly implement the cache management commands. Others have replied that you need to disable the write cache, and while that would be a solution, it is often impossible. Even with bridge chips that do support the cache disable command, some hard drives will not honor it anyway.
Most USB bridges simply lie about when data has been written, which makes it very difficult for a filesystem on top of it to make any guarantees. While it may not happen often, this can have disastrous results, as you have seen.
The copy on write nature of ZFS left it especially vulnerable to broken USB storage, and could easily leave you with a corrupted pool requiring manual intervention and a bit of luck to recover. Thankfully, the recent bits address this, and ZFS is now the only filesystem that I would trust on top of USB storage. Most other filesystems will survive without incident, but at the cost of some silent data corruption.
I know this comment will get lost in the sea of other comments, but my recommendation to you would be a hybrid solution.
Create a small partition (1GB would be overkill) and format it FAT32.
Create another partition for the rest of the drive (or however you please) with your choice of FS (I prefer XFS, personally).
Store the drivers(/utilties) for the FS you chose and store them on the FAT32 drive.
Some popular drivers/utilties for Windows are:
ext2fsd for EXT2 - http://sourceforge.net/projects/ext2fsd/
rfstool for ReiserFS - http://freshmeat.net/projects/rfstool/
ltools for EXT2/EXT2/ReiserFS - http://www2.hs-esslingen.de/~zimmerma/software/ltools.html/
and so on and so forth (a simple google for "[FS] Windows Compatibility" usually works.)
Just my thoughts. :-)
NTFS is recommended several times in response to this article and it troubles me. The next de facto portable filesystem standard is being determined by us in the same way mp3 was made a standard. Ask yourself if you really want to see NTFS become the standard?
No, but that's not really relevant.
Personally I'd rather see a truly open filesystem become standard and I really don't want to see Microsoft have the leverage to make more patent threats. If you agree with me and have some integrity, please recommend an open standard.
Can't reconcile the contradiction in that request. There's no open standard that satisfies the need, so I would have to abandon my integrity to make a dishonest recommendation that I like for its openness. If I want to maintain my integrity, I need to make an honest recommendation that accords with the truth, even if it's an unpleasant truth. For the desired purpose, the sad truth is your best bet is NTFS. I wish that weren't so, I would hope that in a few years some open standard would emerge, but I'd be dishonest to answer with wishful thinking rather than fact.
"Convictions are more dangerous enemies of truth than lies."
+ 1 for build a NAS. Your backup doesn't need to be very portable. I started out with FreeNAS, if you go this route ZFS would be the logical choice. I got pretty frustrated with this so rebuilt my NAS using Ubuntu and EXT3 - it made sense to me to use the native FS for the OS. I currently have 4 x 1TB internal drives in raid 5 (one spare). I use a USB drive to back this up. SMB internally and all the files are accessible by every OS I use. If I need to take files out and about there are good old USB sticks or portable drives. Mine are usually NTFS or FAT 32 because anything can read these - and of course there's always good old FTP on the NAS