Slashdot Mirror
Best Filesystem For External Back-Up Drives?
rufey writes "I've recently embarked on a project to rip my DVD and CD collection to a pair of external USB drives. One drive will be used on a daily basis to access the rips of music and DVDs, as well as store backups of all of my other data. The second drive will be a copy of the first drive, to be synced up on a monthly basis and kept at a different location. The USB drives that I purchased for this are 1 TB in size and came pre-formatted with FAT32. While I can access this filesystem from all of my Windows and Linux machines, there are some limitations." Read on for the rest, and offer your advice on the best filesystem for this application.
"Namely, the file size on a FAT32 filesystem is limited to 4GB (4GB less 1 byte to be technical). I have some files that are well over that size that I want to store, mostly raw DVD video. I'll primarily be using these drives on a Linux-based system, and initially, with a Western Digital Live TV media player. I can access a EXT3 filesystem from both of these, and I'm thinking about reformatting to EXT3. But on Windows, it requires a 3rd party driver to access the EXT3 filesystem. NTFS is an option, but the Linux kernel NTFS drivers (according to the kernel build documentation) only have limited NTFS write support, only being safe to overwrite existing files without changing the file size). The Linux-NTFS project may be able to mitigate my NTFS concerns for Linux, but I haven't had enough experience with it to feel comfortable. At some point I'd like whatever filesystem I use to be accessible to Apple's OS X. With those constraints in mind, which filesystem would be the best to use? I realize that there will always be some compatibility problems with whatever I end up with. But I'd like to minimize these issues by using a filesystem that has the best multi-OS support for both reading and writing, while at the same time supporting large files."
is to stop being so diverse! Pick a platform and stick with it!
Ok, in all seriousness.. here's what you do:
- buy yourself a cheap (~200) box
- hook all your drives to it
- use whatever file system you want (JFS, XFS would be my recommendation)
- share it over your zoo of a network using nfs, samba, etc..
As a bonus, your file server box could double as a media center, and replace your WD TV Live dealie.. (probably not though.. right)
Irregardless, I think you're way better off with this approach vice trying to find the magical widely supported cross platform file system with large file capacity.
You also might want to consider RAID vs. your monthly sync. Yes, RAID isn't a backup.. but for something like this where
restoration would be possible, but just a pain in the ass.. mirrored raid can be a lot more convenient. You can always have
a third external to back up your irreplacable data on a semi-annual basis..
I wouldn't limit myself to a certain filesystem, I'd run a dedicated NAS like FreeNAS and share it over the network via SMB (windows), AFP (apple) and whatever for Linux - all set. Plus as mentioned above, you can run Firefly media server, a bittorrent server, a DAAP server (itunes sharing), etc (all included in FreeNAS. http://freenas.org/) on the same box. And since filesystems don't matter in this config, you can use ZFS to make a RAIDZ pool of your drives. It's what I do now.
fak3r.com
Not if you use a network filesystem, such as Samba and NFS for the Windows and MacOS machines. Then on the Linux fileserver side, use whatever filesystem you want, and any OS can talk to that server.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
As an alternative to an external disk that goes to multiple machines, this might cost some, but perhaps consider a backup server?
The advantages to this setup:
1: The server initiates the backups, and can warn you in case something can't be read.
2: Most backup software stores snapshots, and some deal with the full/incremental/different cycle by using synthetic full backups. This makes restores to a certain point in time pretty easy.
3: More sophisticated backup software allows you to transfer backup sets to another media. This way, you just plug in a drive, do a transfer, and you have an offsite archive.
4: If one of the backup client machines gets hacked or malware installed, existing data stored on backup media cannot be altered.
The disadvantages:
1: You will need an active computer which is significantly more expensive than a hard disk.
2: Amanda/Zmanda for open source, Retrospect, Backup Exec, for commercial. The software costs a hefty chunk of change.
3: You have to make extremely sure that the backup server box is locked down tight. If someone compromises your backup server, they got data of every box you have. If you can, perhaps consider buying a router to put the backup server behind and only allowing the vital ports incoming.
4: Backup servers should have some redundancy for stored data. Because there is so much data stored from multiple boxes, a failure of a drive hurts more than on a normal machine.
5: Restoring a machine may vary in difficulty.
I haven't tried it but it looks like a good idea. http://www.cyberguys.com/product-details/?productid=36218&sk=MC71419
Format it ext3 and then share it SMB for any OS.
You are lucky, Ed Gruberman. Few novices experience so much of Ti Kwan Leep so soon.
I see quite many people here recommending ext3. Oh my. ext3 sucks for large files,
which is exactly what the submitter wants to use his setup for. Look into the crazy structures
("double indirect blocks") it uses. He should go with an FS that has sane data structures with
files >>4GB.
That kills most of the choices and leaves XFS, ext4, ZFS (only worth it if not used via FUSE,
i.e. in Solaris), and a couple more obscure ones.
I second the "forget OS portability, use a server" suggestion. There's great low-power, low-cost
hardware for this nowadays.
Did you "zpool export tank" before you moved the drives over? If not, then the FreeBSD box saw the drives but said to itself "Those drive don't belong to me!" I've done a migration from FreeBSD to Solaris, but always with full drive devices ie /dev/ad0 - thus ignoring partitions tables and other such cruddyness.
Moneyed corporations, non-working 'poor' and criminal prisoners are turning productive citizens into tax-slaves.
I know this comment will get lost in the sea of other comments, but my recommendation to you would be a hybrid solution.
Create a small partition (1GB would be overkill) and format it FAT32.
Create another partition for the rest of the drive (or however you please) with your choice of FS (I prefer XFS, personally).
Store the drivers(/utilties) for the FS you chose and store them on the FAT32 drive.
Some popular drivers/utilties for Windows are:
ext2fsd for EXT2 - http://sourceforge.net/projects/ext2fsd/
rfstool for ReiserFS - http://freshmeat.net/projects/rfstool/
ltools for EXT2/EXT2/ReiserFS - http://www2.hs-esslingen.de/~zimmerma/software/ltools.html/
and so on and so forth (a simple google for "[FS] Windows Compatibility" usually works.)
Just my thoughts. :-)