Slashdot Mirror
Best Filesystem For External Back-Up Drives?
rufey writes "I've recently embarked on a project to rip my DVD and CD collection to a pair of external USB drives. One drive will be used on a daily basis to access the rips of music and DVDs, as well as store backups of all of my other data. The second drive will be a copy of the first drive, to be synced up on a monthly basis and kept at a different location. The USB drives that I purchased for this are 1 TB in size and came pre-formatted with FAT32. While I can access this filesystem from all of my Windows and Linux machines, there are some limitations." Read on for the rest, and offer your advice on the best filesystem for this application.
"Namely, the file size on a FAT32 filesystem is limited to 4GB (4GB less 1 byte to be technical). I have some files that are well over that size that I want to store, mostly raw DVD video. I'll primarily be using these drives on a Linux-based system, and initially, with a Western Digital Live TV media player. I can access a EXT3 filesystem from both of these, and I'm thinking about reformatting to EXT3. But on Windows, it requires a 3rd party driver to access the EXT3 filesystem. NTFS is an option, but the Linux kernel NTFS drivers (according to the kernel build documentation) only have limited NTFS write support, only being safe to overwrite existing files without changing the file size). The Linux-NTFS project may be able to mitigate my NTFS concerns for Linux, but I haven't had enough experience with it to feel comfortable. At some point I'd like whatever filesystem I use to be accessible to Apple's OS X. With those constraints in mind, which filesystem would be the best to use? I realize that there will always be some compatibility problems with whatever I end up with. But I'd like to minimize these issues by using a filesystem that has the best multi-OS support for both reading and writing, while at the same time supporting large files."
It is open, and the driver is available for windows. And I don't see it just disappearing, so nothing can read it.
is to stop being so diverse! Pick a platform and stick with it!
Ok, in all seriousness.. here's what you do:
- buy yourself a cheap (~200) box
- hook all your drives to it
- use whatever file system you want (JFS, XFS would be my recommendation)
- share it over your zoo of a network using nfs, samba, etc..
As a bonus, your file server box could double as a media center, and replace your WD TV Live dealie.. (probably not though.. right)
Irregardless, I think you're way better off with this approach vice trying to find the magical widely supported cross platform file system with large file capacity.
You also might want to consider RAID vs. your monthly sync. Yes, RAID isn't a backup.. but for something like this where
restoration would be possible, but just a pain in the ass.. mirrored raid can be a lot more convenient. You can always have
a third external to back up your irreplacable data on a semi-annual basis..
If you're like me, you won't be happy with the compromises you have to make when picking a multi-platform filesystem. I'd outline them, but you've done a great job of doing so above. So, what to do?
Get thee a cheap, cheap Linux box, format your drives EXT3, and all other machines access over the network. It's the only way you'll get the interoperability you want, without making compromises on max file size, cluster sizes, etc.
THat's what I use here. for MAC, there's NTFS-3G (free) or Paragon ($ but faster on writes).
I use NTFS on both my machines (Win/OS X/Linux) without any problems.
NTFS-3G is also available for Linux.
I've got better things to do tonight than die.
Have you been living under a rock? NTFS has been writable on Linux for a long time now, using NTFS-3g - most distros come pre-bundled with it nowadays, and there's a Mac version as well.
VLC can play rar-compressed-splitted files beautifully, So 4GB is not a very big problem
...ReiserFS. I hear it's killer.
I have discovered a truly marvelous
I wouldn't limit myself to a certain filesystem, I'd run a dedicated NAS like FreeNAS and share it over the network via SMB (windows), AFP (apple) and whatever for Linux - all set. Plus as mentioned above, you can run Firefly media server, a bittorrent server, a DAAP server (itunes sharing), etc (all included in FreeNAS. http://freenas.org/) on the same box. And since filesystems don't matter in this config, you can use ZFS to make a RAIDZ pool of your drives. It's what I do now.
fak3r.com
Via FUSE you'll get consistent features and useability across all 3 OSes. Of course moving zfs drives between those OSes isn't something I've tried, but in theory it should work fine.
Not what your asking for, but Id put a FBSD samba server up with ZFS drives. You can still mount them on other OSes later if need be via FUSE.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
Posting anon cuz this is TOTALLY offtopic and I hate getting negative mods, but Wikipedia sums it up pretty well in the first sentence:
"Irregardless is a term meaning regardless or irrespective, which has caused controversy since it first appeared in the early twentieth century. It is generally listed in dictionaries as 'incorrect' or 'nonstandard'."
I backup to portable USB hard disks. My backup machine is my eeepc 701. It runs ubuntu. I use this machine because it has fast USB and wifi interfaces. I have written a short shell script which runs on the eeepc. It uses rsync through ssh to copy user data from all the machines in the house to the external disk. I ignore the single windows machine in the house. If its user wants it backed up they can store their files on the server.
I initially tried backing up through a workstation runing netbsd but I found the USB interface to be too slow, by many orders of magnitude.
http://michaelsmith.id.au
Honestly, if FAT32 won't do what you need, NTFS is pretty much where you'll need to go. NTFS-3g gives you stable read/write capability on Linux and OS X as a FUSE driver; in fact, many distributions have NTFS-3g in their repositories. There's also native NTFS write support in Snow Leopard if you want to risk turning it on. I personally haven't had any issues with it, but some people have encountered file corruption when using it, so you might want to be wary. It is worth noting, however, that many embedded devices won't read anything other than FAT. If you plan on hooking this drive up to, say, a DVD player to show pictures, NTFS won't work for you.
Like it or not, Microsoft file systems are the lingua franca of file transfer on portable drives these days, merely due to the installed base of Windows computers.
The Freelance Wizard
"Have you been living under a rock? NTFS has been writable on Linux for a long time now..."
Oh yeah, because the determining factor of one's general awareness is whether or not one knows that NTFS is writeable on linux.
I knew that, but I certainly didn't hear the announcement on a TV ad during "House".
"ir" HTH! HAND!
I have ~6TB on external USB drives and I've been doing this for a few years now. I have a few words of caution about NTFS. If you get an USB drive that for example spins down or if you turn your USB drive off without properly dismounting it (or if Windows crashes), you might see this line:
Delayed write failed!
And on two occasions that meant that Windows fucked up the file allocation table or whatever it's called under NTFS and I lost the _entire_ disk.
Windows loves getting its fingers into that table whenever you mount a USB filesystem. It's not like it tries to keep its write cache empty. Nooo.. every file access needs to be continuously recorded in that thing.
Anyway, be careful when you use NTFS on a USB drive. Alternatively use EXT3, which you can still mount under Windows using:
http://www.ext2fsd.com/
(Note that these experiences are under Windows XP, I have no clue if Vista or 7 does any better, I assume not.)
Not if you use a network filesystem, such as Samba and NFS for the Windows and MacOS machines. Then on the Linux fileserver side, use whatever filesystem you want, and any OS can talk to that server.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
yeah.. everybody and their mother has had confidence that the ntfs Linux driver will improve, but the same can be said for windows as an operating system ;)...
A cheap box running FreeBSD with ZFS, NFS and Samba.
I use NTFS on mine.
Windows obviously works, linux works well with NTFS-3G and I believe you can get NTFS-3G to work in OS X via macfuse.
You want something that will be read by your Linux, Windows and OS X machines? um, only one option I can see and thats FAT32. Any of their own systems, such as NTFS, get you only browsable directories by one or two of the other boxes.
Ave Molech Setting
If proprietary filesystems are on the table, how about VxFS ?
Another possibility is to use FAT with cross-platform backup software. Maybe you don't need a filesystem at all: if this really is for backups... why not just create lots of extended partitions on the device and use TAR ?
AKA tar cf /dev/sdbXX -V 'VOLUME_A' /backup
That's crude and hard to keep organized, but also effective. Also, Some proprietary backup products that will work on a FAT filesystem, and not require large file support, even to backup large files
Or utilize a tool such as WINRAR that allows you to "split" a RAR file across multiple archives in chunks of a certain size, then store these files on a FAT filesystem.
FAT is the most cross-platform, oldest. But has known issues with fragmentation, and lack of journaling, effects reliability.
You could divide your backup volume into 2 partitions: one DOS/FAT partition with the bootable image and files required to 'load a virtual machine' that can see the files on the other partition in the preferred data format such as ZFS or FFS (e.g. pre-allocated eager zeroed thick VMware VMDK with 'split into 2gb files' enabled).
Then you just make sure the system you plug the drive into can boot a VM, with your "backup/file access environment"
It's a shocking bad state to be in in 2010... Something needs sorting. You can try UDF, but there are so many revisions, finding one that works for everything is a challenge too.
I've found tapes to be brilliant for backups, especially if you want to do a regular backup weekly or monthly and have access to several backups from the past. Granted, it is very expensive for an individual or family, I've only had good experiences with it.
As an alternative to an external disk that goes to multiple machines, this might cost some, but perhaps consider a backup server?
The advantages to this setup:
1: The server initiates the backups, and can warn you in case something can't be read.
2: Most backup software stores snapshots, and some deal with the full/incremental/different cycle by using synthetic full backups. This makes restores to a certain point in time pretty easy.
3: More sophisticated backup software allows you to transfer backup sets to another media. This way, you just plug in a drive, do a transfer, and you have an offsite archive.
4: If one of the backup client machines gets hacked or malware installed, existing data stored on backup media cannot be altered.
The disadvantages:
1: You will need an active computer which is significantly more expensive than a hard disk.
2: Amanda/Zmanda for open source, Retrospect, Backup Exec, for commercial. The software costs a hefty chunk of change.
3: You have to make extremely sure that the backup server box is locked down tight. If someone compromises your backup server, they got data of every box you have. If you can, perhaps consider buying a router to put the backup server behind and only allowing the vital ports incoming.
4: Backup servers should have some redundancy for stored data. Because there is so much data stored from multiple boxes, a failure of a drive hurts more than on a normal machine.
5: Restoring a machine may vary in difficulty.
I bought a LaCie 1 gig drive with an ethernet connection a few years ago. Dunno the file system, prolly Linux with Samba. Just plug it into network.
Everything connects to it including the last DOS 3.1 box needed for a legacy database.
And it was only $200. You want mirroring? Buy a second and copy over the first with a batch file whenever you get nervous.
Replace the silly little WD TV Live media player with a mITX system that's about the same size. Install Linux and XBMC and be done with it. You'll have the best possible media player on the planet, as much storage space in any configuration you want and the ability to expand everything when the time comes. No hassle, you'll have constant online backups available and you'll have a killer always-on media center.
IIRC, NTFS is a descendent of something called HPFS, which is what IBM developed for OS/2. At least as recently as Win2K, Windows would mount and use HPFS partitons, and also I recall that Linux could read/write that as well. Look into that.
If you need Linux/Mac/Windows interoperability then we recommend NTFS for both Linux and Mac users. Instead of the old NTFS kernel driver you may want to check our open source NTFS-3G. It has read/write, and tons of options:
http://www.tuxera.com/community/ntfs-3g-advanced/
If you need just high-performance NTFS read/write, this is our offering for Mac users:
http://www.tuxera.com/products/tuxera-ntfs-for-mac/
If you need high-performance for a commercial Linux application or device, you may want to check this:
http://www.tuxera.com/products/tuxera-ntfs-commercial/performance/
Regards,
Mikko Välimäki
CEO, Tuxera Ltd
Make sure you have write cache disabled! There's even a warning in Windows that having write cache on will cause data loss in this kind of case. I have it off and I've never encountered this issue. It should be set to disabled by default.
2 disks, redundancy, heck buy a cold spare HD and you will be protected better than two USB drives
It might if irregardless was actually a word.
Normalcy didn't exist until the 1850s, even though normality was 150 years older. (I'd never heard of normalcy until U.S. History class...)
"I don't know, therefore Aliens" Wafflebox1
Works just fine on the Amiga...
Ooh, wrong century... sorry
Personally I like Openfiler. It can be picky about the hardware though. With that said, the speed is great, and I can mount iscsi on linux and windows. Has been stable as hell for me to boot.
when logic fails, bullshit prevails
Looks to me like HFS is the way to go since there are good solutions for all three platforms for HFS.
I go with ext3 for this personally. NTFS doesn't store *nix style filesystem permissions, and causes various other issues with you Linux systems. With ext3, you can store all your files with all of your permissions intact, the filesystem is mature and trustworthy, and you can still access all of the files from any operating system by simply connecting the drives to a dedicated fileserver machine (an older computer or small device works perfectly for this). Simply share your files via NFS, Samba and ftp (if you need ftp access for something like xmbc). Having a dedicated machine for this means you can also script your replication to the secondary drive, so that you only have to attach the drive for the mirroring process to take place.
This is the solution I've been using for about four years now, and it works great for me.
One word : FreeNAS.
Its been stated here repeatedly. You want to go NTFS for all the listed reasons, or ZFS.
I set up my own ZFS NAS server, but yeah, up to you.
Do NOT go FAT32 under ANY circumstances.
You can tell how powerful someone is by the magnitude of the crime they can commit and be able to get away with.
Just use NTFS. You say you'll be using a WD TV Live! (which is a wonderful device, I must say), which supports NTFS and can act as a fileserver on your network. So just leave your daily drive connected to the WD TV Live!, push your backups to it over the network, and then do your monthly backups off that as you planned. No reason to bother with anything more complex than that.
Considering DVD files are typically split at the 1 GB mark, you should have no issues with DVD. Assuming you have other files that break the 4GB barrier however, leaves you with EXT3, HPFS, or NTFS. The EXT3 support for OS X is supposed to be there using the SourceForge EXT2 driver, but I never managed to get them to mount. NTFS support is good on the OS X side, seems to be good on Linux and of course on Windows.
I'd go with NTFS using NTFS-3G as it gives read/write support on all 3 OS's and is open source ( http://sourceforge.net/projects/ntfs-3g/ )
I have been through exactly the same thing, and NTFS is the way to go. My main machine is a Mac, but I format all my USB drives as NFTS (through NTFS-3G FUSE). Though this is slower than formating as FAT or HFS+, it is so much more portable, it wins out. Windows machines can read them. My WD TV box can read them. My Mac can read them. They can store more than 4G per file (indeed, I have all my DVD isos on them). Ok, they can't be read on other macs without installing NTFS-3G, but I've yet to need that. For me, I've never had any problem with NTFS-3G , it's extremely reliable.
Enough said, next topic.
Why is the Windows's state of support third party file systems so bad anyway? Why does the Linux kernel supports so many file systems and Windows basically only fat32 and ntfs?
I thought because the ABI of the Windows kernel is stable it is so easy to develop drivers for it.
Oh and while am at it, when Windows will have LVM2 support? I can resize every partition on my Laptop as I wish, only the Windows partition is a static block (a too large block, btw, unable to be resized and used where the space is needed).
http://www.mueller-public.de - My site http://www.anr-institute.com/ - Advanced Natural Research Institute
The solution ... is a good rant. God, I love a good, heartfelt, I-don't-give-a-f*ck-how-crazy-it-makes-me-look rant. I mean, if we were to database all of the word misusages, mispelings, verbing of nouns and nouning of verbs, if we crammed all of the unGodly linguistic trashfulness of /. into a forward-looking, high-concept, low-brow PowerPoint® slideshow we could ... probably make multimediavt go nonlinear in a very exponential, straightforward way. It would exciting! It would cause fireworks across the net!
By God! It would be ... impactful!
Let's do it!
My backup has been worked good for the last few years. I had a large windows tower filled with hard drives...
* One of my HDD was encrypted using Truecrypt.
* A scheduled Acronis task would image my OS drive to the encrypted drive.
* I have another encrypted HDD in one of these KINGWIN KF-91-BK trays (http://www.newegg.com/Product/Product.aspx?Item=N82E16817121172)
* After Acronis was done, I would kick off a little batch file to mount the tray drive (which was encrypted) and run SyncBack Pro to mirror the encrypted source. Then dismount.
I just bought a few extra trays. Each month I sync all the other HDD in the system (including the encrypted drive) for my off-site backup. One flaw in my system was if one of the other "data drives" crashed, I would loose at most a month of changes. They were just music/videos and didn't change very much. I accepted this fault.
I out grew the tower and recently built a cheap unRaid box. Most if it should work the same, except now I have redundancy for all my drives in the case of a crash.
Use NTFS and then install the Fuse ntfs-3g driver. This works for read write in linux and OSX and makes it easy. We all know that Windows supports Microsoft only and that leaves us to configure the other systems we have. Your distro should have Fuse as a precomplied package and probable the ntfs-3g driver as well. As for OSX you can find it by searching for "mac fuse" in your favorite search engine.
It doesn't even come close. (I linked to one small part of the article. Please read the whole thing before coming to the realization that I am correct.)
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
I've used NTFS-3G successfully. http://www.tuxera.com/community/ntfs-3g-download/
But the nfsmount people have this to say: "The ntfs-3g driver is an obsolete fork of ntfsmount. Use ntfsmount from ntfsprogs-2.0.0"
http://www.linux-ntfs.org/doku.php?id=ntfsmount
Who is correct???
"I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
NTFS-3G, which should come standard in most distros, should be able to read and write NTFS perfectly. It's considered very stable. That said, my personal solution to this problem was to use EXT2 and install EXT2IFS on my windows machines. I had a small FAT32 partition on the USB disk with the EXT2 driver installers for Windows and MacOS, so if I ever need to read it on another computer I don't have to download anything.
I format my external USB drives to ext3. Most of my machines are Linux anyway, and I can always plug the USB drive into my storage server and backup over Samba to any kind of drive supported by the storage server.
ext3 is pretty much stable and well understood. It just works. That's what I want for backup drives.
And my netbook has Ubuntu Linux on it, and ext3 performs well on the external USB drive there. I haven't tested NTFS over FUSE on the netbook, but I wonder about CPU overhead on the little Atom chip: it might be a little bit slow.
If you want a drive you can take over to your friend's house, and your friend just runs Windows or a Mac, then by all means NTFS.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
Ignore this thread. Once you get past all the first post trolls, the rest of it mainly contains bickering over whether "irregardless" is a word or not. Slashdot at its finest, I know. So I'm jumping in at the top of the thread to warn you and save you the time.
As for which filesystem to use: It's a shitfight at the end of the day. OSS-NTFS is reverse engineered so you can never trust that it really is complete, nor that Microsoft will never bend/break it in the future. FAT32 sucks but is unquestionably the lowest common denominator. I like to use Ext2 for getting big files from Linux to Windows (I transfer a lot of DV files), but you have to install something on the Windows/Mac/etc. to use that. HFSplus support on Linux is fine, you just need to disable the journal on the Mac first.
Like I said, it's a shitfight. Learn the pros and cons of the main filesystems: NTFS, Ext-2/3 family, HFS, FAT32. Break them out as each situation merits. But with regard to NTFS and the specs changing: Microsoft have done it before and they'll do it again, just ask Norton Utilities. While my Windows PC is a very important part of my operation (I edit all my video on it), I don't use Windows for enough other things to justify using NTFS unless I'm forced to.
Install a tiny debian virtual machine on your windows PC that can host the USB drive as a network share for the times when you have to use it via Windows. There maybe some kind of Virtual-box-system-in-an-exe type thing where you could have it on a FAT32 partition at the start of the disk so you could run it on any windows PC without the need to pre configure a VM. Depends on your situation.
It's stable, it works, it uses Fuse so you don't have to worry about Kernel support.
Just go get it.
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
I haven't tried it but it looks like a good idea. http://www.cyberguys.com/product-details/?productid=36218&sk=MC71419
Format it ext3 and then share it SMB for any OS.
You are lucky, Ed Gruberman. Few novices experience so much of Ti Kwan Leep so soon.
I see quite many people here recommending ext3. Oh my. ext3 sucks for large files,
which is exactly what the submitter wants to use his setup for. Look into the crazy structures
("double indirect blocks") it uses. He should go with an FS that has sane data structures with
files >>4GB.
That kills most of the choices and leaves XFS, ext4, ZFS (only worth it if not used via FUSE,
i.e. in Solaris), and a couple more obscure ones.
I second the "forget OS portability, use a server" suggestion. There's great low-power, low-cost
hardware for this nowadays.
I would recommend NTFS. I would also recommend that you get an open source program and compress the dvd's to approx. 700MB for up to 90 minutes and 1.4GB over 90 mins. H264/ac3 or Xvid are good codecs to use. Batch schedule the movies and let the system compress all night when you don't need the system. The reason for this is because dvd's take up a lot of wasted space because of the format. A compressed movie takes up far less space. I can store about 1000 movies on a 1TB drive and if a dvd is ever needed, I can recode a dvd. There is not much quality difference between a compressed movie and a dvd file. You can also convert the dvd's to .mov files but apple can read most H264 files or mp4 files. Many dvd players can read Xvid or Dvix files and others without the need for a dvd file.
For cd's, If you plan to burn several copies, then rip the cd's directly to .wav or .flac (lossless) files. mp3 files are like .jpg image files such that they lose a little quality/data each
time they are copied whereas lossless .wav or .flac files DO NOT regardless of how many times they are copied.
English is a living breathing thing; if he or she likes iregardless, so be it; words change; unique no longer means unique, but unusual
As Orwell once said, regarding split infinitives, its better to break any rule then do something barbarous..
For my backup, I use an external firewire drive that I leave turned off, except once per month, when I turn it on, mount it (linux system), and run rsync between it and another drive (my home directories). Runs very slick. I've also used Rsync to recover data off of a dying drive. I could only get data off of it for about 10 minutes (and it had to be cold). Rsync worked like a charm. I got alll the data off after a few hours (about 85 GB, 10 minutes at a time), followed by 30 minutes of cooling down. Rsync worked like a charm. I would use Reiserfs to handle the large data size (easily handles 1TB and 100 TB filesystems. I can also add encryption to it so that if the drive gets stolen, then yes, they will have to crack skipjack in order to get at the data. Its not impossible, but, good luck with that.
If your Windows PC(s) are all networked with your Linux PC, then just format it Ext3, share it out with Samba, and let Windows read it from over the network.
Dear AP,
I was pondering with a very similiar problem two years ago, when I bought my first terabyte-class external USB hard drive to my home server (old 800 MHz Duron). I was thinking exactly like you are thinking now, and decided FAT-32 would be the way to go. MISTAKE. Four important things stand out, that I want you to read:
Okay, those are my four vocal points. They could be in any order, because all of them are equally important reasons NOT to choose FAT-32! As it happened, after using the 0.5TB drive for 6 months with FAT-32, I bought more space (a new drive). This time there was no question about the filesystem. I made a small, few-sector long 200 MB FAT-32 partition to the beginning of the drive and downloaded all the latest Win32 EXT2 drivers there from different vendors, just for the really unplausible situation that I would ever want to mount these drives in Windows. Then I just made the rest Ext3. And, I am REALLY satisfied with the decision! Ext3 just work so sparklingly faster and better with Linux than FAT-32 ever does. Since then I have bought one more drive and did the same 200MB + 1TB thing. I will probably never use these drives in Windows, but it gives me a warm feeling to the heart that there's always a way if I should, even if the computer doesn't have an Internet connection.
Oh, one thing I forgot to mention: get a file server! It makes your life so much easier. Nowadays I am running a desktop computer with a 60 GB SSD drive and no HDD at a
I am surprised that nobody has mentioned it, but what about a server running AFS (Andrew File System).
It has some great features:
Scalable - you can add as many servers as you want
Secure - it uses kerberos by default
Fast - local caches are kept until another client needs them.
Cross platform - openAFS supports pretty much every platform in existence.
Why is no one recommending ext4? Because of it lacking windows support? Is there any reason why ext4 should not be used in a simple raid5 server?
Bravo!
Now how about applying it to all the other posts in this fucking useless thread that are off-topic?
Comment of the year
I would have agreed with you a couple years ago, but ntfs-3g has become quite good - zero problems reading and writing large amounts here, and I use it for work almost daily.
WHO NEEDS SHIFT WHEN YOU HAVE CAPSLOCK/ DAMN1
I'm not sure if this is possible, just throwing it out there... Could you install a virtualized OS instance on all these hosts, with VirtualBox or VMWare, on each computer? If so AND you were able to access the raw drives and the hosts files from the vhost (with some magic that I'm not aware of), you could use a filesystem native to the vhost but not to the metal host.
You're so torn between formats, why don't you just partition the external drive with like ten partitions and format each one with a different format -- that way you don't have to choose!
The big problem with ReiserFS: Vendor lock-in.
This is the perfect application for FreeNAS. In additional to all the benefits listed above, you've got the options of using rsync to maintain your off-site drive.
Use a journaled file system - I don't care much which you choose. I prefer JFS or XFS and don't think EXT4 is ready for trusted use yet. For the live data, I'd add Linux software RAID1 or RAID5 or RAID6 protection. mdadm has never let me down even with corrupted disks. Not a single bit has been lost in 5 years. For the backup drive ... it doesn't matter too much, but I'd stay journaled.
For media, that doesn't change much, I'd use rsync to mirror the disk file systems efficiently.
ZFS with zsend would be more efficient, but ZFS has that problem - no Linux kernel support. As much as I love ZFS, I'm not going to run a new OS on my home network just to use it.
If you really care about this data, consider adding some par2 files to ensure you can correct corruption over time - you know, bit rot (which really does exist). ZFS has data validation built-in.
At least you aren't planning to store this on DVDs for 10 years and believe all the bits will be there when you come back.
XFS was designed with media streaming in mind -- and designed for large file, high performance. It had a defragmenter to keep disks in optimal condition before Windows98 had come out (was one at the request of a large, customer who had an especially pathological case -- before that there was normally not considered a need for it).
Files can be 'normal', have up to and addition 256K of resource-fork related into (extended attribute info), AND you can have a real-time section that can allow for completely bypassing the file system. It was sufficiently fast for video even back when disks were 1/10th the speed they are now.
On it's native OS, it could handle multiple streamed data to the same disk and keep it separate by allocating the separate channels out of disparate allocation groups on disk. I don't know how that works on linux. Unfornately, on linux even under x64, file block sizes AFAIK, are still limited to 4K. XFS has a 64K limit, but under linux is hamstrung to 4k. Of course Windows NT allows 64K block sizes. But not linux...hmmm....very weird. XFS minimizes impact of linux's tiny allocation block size by using a extents which can be at least 256k -- but believe the actual limit is in megabytes. Been a while since I read that stuff...
Of course -- not to be linux centric, but have heard ZFS is pretty good, but no idea of how it compares for anything.
my 2 cents...
I use a similar backup system and backup schedule to an external USB drive as well. I used NTFS for the filesystem but I mount Truecrypt volumes on that file system to hold my backup data. If the drive fails and I have to send it back for warranty purposes or if someone steals it, at least I know my backups were resonably protected with some type of encryption. Another benefit is I can move my entire backup set to a new drive by simply copying that truecrypt volume file to the new destination.
NTFS3G for Linux seems to work perfectly for NTFS file systems (read AND write); and, unless you're creating ISO images of your raw DVD video, the VIDEO_TS folders should have files that are just under 1GB each (for DVD filesystem compatibility), making FAT32 usable (or are you saving raw capture from a video camera, making huge uncompressed files that are >4GB?)
The fundamental problem lies in USB bridge chips which do not properly implement the cache management commands. Others have replied that you need to disable the write cache, and while that would be a solution, it is often impossible. Even with bridge chips that do support the cache disable command, some hard drives will not honor it anyway.
Most USB bridges simply lie about when data has been written, which makes it very difficult for a filesystem on top of it to make any guarantees. While it may not happen often, this can have disastrous results, as you have seen.
The copy on write nature of ZFS left it especially vulnerable to broken USB storage, and could easily leave you with a corrupted pool requiring manual intervention and a bit of luck to recover. Thankfully, the recent bits address this, and ZFS is now the only filesystem that I would trust on top of USB storage. Most other filesystems will survive without incident, but at the cost of some silent data corruption.
I have a LOT of my DVD collection rip'd to my computer drives. I use 1.5GB esata hard drives formatted ext3 for Linux and have a linux file server running Samba so all my Linux, Mac, and Windows machines can access the ISO images (mount as virtual DVD's) from anywhere in the house. Works great, costs very little ($0.10 USD / GB == $0.40-0.90 / DVD), and can easily be copied to a laptop for travel.
I know this comment will get lost in the sea of other comments, but my recommendation to you would be a hybrid solution.
Create a small partition (1GB would be overkill) and format it FAT32.
Create another partition for the rest of the drive (or however you please) with your choice of FS (I prefer XFS, personally).
Store the drivers(/utilties) for the FS you chose and store them on the FAT32 drive.
Some popular drivers/utilties for Windows are:
ext2fsd for EXT2 - http://sourceforge.net/projects/ext2fsd/
rfstool for ReiserFS - http://freshmeat.net/projects/rfstool/
ltools for EXT2/EXT2/ReiserFS - http://www2.hs-esslingen.de/~zimmerma/software/ltools.html/
and so on and so forth (a simple google for "[FS] Windows Compatibility" usually works.)
Just my thoughts. :-)
Just a tip, it's really disasterous when you don't realize that data has been corrupted, and external drives can be a wee bit more susceptible than we might hope... Try setting up checksumming in appropriate places, and before deleting older versions of things, just check what you "think" you still have... (dvdsig.exe is a prog I have seen around for this purpose... - or create some scripts!)
That's interesting, I didn't know that. I do know that a certain E-Sata 'hot swappable' card I installed a while ago *COULD NOT* be set to 'disable write caching' in XP. It was a mounted volume, and Damnit We're Going To Treat it As Such. I was a bit pissed about that. You know *MAYBE* I bought a hot-swappable E-Sata card so I could use it with a removable disk? Perhaps!? Thankfully there's a 3rd party program that flushes mounted volumes so you can 'eject' them - and Win7 handles it natively. But srsly.
The option being disabled makes a lot more sense now. I just kinda assumed (doh!) that it was Windows handling the caching.
Works for me perfectly on Linux and Windows on an external USB-Harddisk for 3 years now.
The idea of using a USB drive for any sort of large storage gives me the willies for the reason you list, which is extremely important, as well as some others:
-USB bridges rarely pass any SMART data back to the host, which means that if it starts to fail you won't know until it's already lost data. How much data? You'll have no idea. Well, you could if you used ZFS or other checksumming, but even then you're not going to know what wrong until data is gone.
-Drives in the 1TB range are going to be regular internal drives with a USB bridge on them. Ever notice how external drives normally have a much shorter warranty on them than the internal versions of the same basic disk? That's because moving a desktop drive around is really bad for it. They just don't last very well in that situation, thus the derated warranty and lower expected lifetime.
-Drive failure rates are so bad, and the time it takes to make a full copy of a drive this big so long over USB, that you're way too exposed to losing two drives. If you care about your data at all, ideally you'd want a RAID-1 pair of internal drives in a server system. It's reasonable in that situation to make an off-site backup you sync periodically, but you don't want a portable drive to be your only primary copy.
Note that the first couple of these issues, the ones related to the crappy USB chipsets, can be avoided if you get an external drive that connects via eSATA. Then you get SMART, no write cache fiasco, much better situation. But it's still a desktop drive that's very likely to fail when used this way.
True, I have had zero problems with Windows XP / ntfs and Linux in recent years. But what about Windows 7's ntfs? I've upgraded from XP to Windows 7 and I have found that Ubunut Karmic has problems accessing *some* of the directories on the ntfs-filesystems.
Another thing, will windows 7 "enhance" my external HD's ntfs so that I won't be able to acccess it from Linux? The external HD is mostly a storage/backup disk but I use it occasionally to transfer files to Windows 7.
(founded 95,000,000 yrs ago, very space opera)
...there your heart is also.
Just something to think about. :)
"Those who consume the bulk of goods are those who make them. We must never forget this secret of our prosperity."
Use dd and write down all your byte offsets. Seriously though, go for EXT3 and make sure you don't *write* anything from Windows if you don't trust the drivers. Later on you can do an in-place upgrade to btrfs if you want.
Great feature would be data chacksums - for media collection on drive, I would like to get warning if there is any data loss. btrfs has this feature (between many others), when it is a bit more stable, than I will use it in similar scanario.
I ended up doing as most in this thread did. Networking.
I bought a BubbaII from http://www.excito.com/ its a small fan less linux box with 2x usb, 2x ethernet, and 2x extSata.
NB: NICs are gigEthernet, but they perform substantially slower than one expects. This according to the manufacturer, is by design to keen the temperature at a resonable level to accommodate the fan less design.
When in danger, whewn in doubt! Run in circles, scream and shout!
I've been using this for years, but I do get solid lockups with very large files (over 4GB) which has made me very nervous.. Any idea where I screw up?
Desktop and laptop are still WinXP (home) until I make up my mind what I'll replace it with (it's narrowed down to OpenSuSE or Mandriva), but for some use a Windows desktop is still more suitable (I'd love Paint.NET to run on Linux, for instance).
Insert
Create different partitions on external drive, one for example EXT3 and other FAT32. EXT3 would be large and FAT32 would be very small couple of megabytes. On smaller partition you would have EXT3 drivers for Windows machines. With this you would be able to access EXT3 from any machine. You could also try different file systems in same way.
WAFL
I am the unwilling control for my Origin.
-1 is not descriptive enough.
Write something like:
-1, freedom-freak or
-1, rms-fanboy or
-1, yeah-just-the-food
Ext file systems are almost useless on windows because of their case sensitivity. Hard-coded file names in applications are often different from those files they refering to. So windows require at least one drive without case sensitive file system where applications and system would be stored.
A few friends asked me to share my backup script: script, so I did.
Enjoy
Or buy a $149 4 disk eSATA/USB external array from Addonics and do software RAID5. 4 x 320GB is about 1TB usable. Then use that 2nd 1TB disk to rsync weekly however you like, since we all know that RAID is not a backup. Can't say that I'd go much bigger on the RAID, since going bigger causes backup issues. I suppose you could put in bigger drives and limit each partition to 1.5TB so that external 1.5TB disks can be used to backup everything up.
Personally, I like having 30 days worth of backups - 1 full with 29 incrementals. The full backup is always from very early this morning.
Further, there's no need to waste all that storage with MPEG2 - convert to the most efficient video format that all ur devices can handle and avoid proprietary codecs since they can change with a little upgrade and break all your devices. I've been using xvid for 6 years for this reason. Sadly, my MediaGate doesn't do HD or mkv or HDMI, but the WD TV Live I'm getting for Xmas will.
In my opinion, external backup drives cannot be trusted. You could work around this by mirroring the drives yourself, but I think that I have a better solution.
Use ZFS.
Here's how I plan to backup Windows files onto ZFS external drives.
1. Install Virtualbox to run OpenSolaris which will manage the ZFS drives.
2. Get a distro of OpenSolaris that supports dedup. The first distro with all of that was 128a so google for this filename "osol-dev-128a-x86.iso" or download a more recent OpenSolaris.
3. Get some USB hubs, one for each of your USB ports.
4. Plug in a bunch of USB drives. I am using eight 8Gb thumbdrives but that only gets me 48Gb. Use spinning hard drives to get more storage.
5. Decide on RAIDZ2 or RAID 1+0 (a stripe of mirrors). If you are using drives bigger than 500Gb then you probably should go for RAID 1+0 aka RAID 10. Be careful, some advice on the net gets this backwards. I am using RAIDZ2 with 2 parity drives which is why my eight drives only give me 6*8=48Gb of storage.
6. Install OpenSolaris in Virtualbox. If you are an uberhacker like me, do a console install on a 2G CF card. If not, then choose a 6Gb virtual drive and do the standard GUI install.
7 Set up your ZFS pool on the USB drives, make some filesystems and export them as CIFS shares or make some virtual devices and export them as iSCSI.
8. On Windows, copy your files to the network drive.
9. In the virtual OpenSolaris machine run a ZFS scrub.
10. Shutdown OpenSolaris and pack away the drives until the next backup session.
You now have a reasonably secure backup which can survive the loss of one, or maybe two drives depending on how you configured it. And if a sector on a drive goes bad, ZFS scrub will recover the data. Normally, an external drive failure means either total loss of the data, or at best, loss of some files. And an external drive may have already lost some of your files but you won't know it until you try to use them years from now. On other filesystems there is no equivalent of the ZFS scrub with sector-based checksums to automatically recover bad blocks.
Yes, I know this solution is a bit techy, but expect someone to package this up as a neat backup-in-a-box solution. I believe that you can run Virtualbox on Windows in a headless mode so the customer would never know that their Fort-Knox grade backup application is actually OpenSolaris in a VM.
I hate to burst everyone's bubble here, I work in enterprise sector of IT, and USB disks are NEVER used as a backup. Simply put, they fail far too frequently. You'll be plagued with bus resets, crappy USB firmware on the drive, and a lack of a true archival medium.
Hard disks are not meant to sit on a shelf and reliably store data for many years. If you truly want a BACKUP that will be usable for years to come, you should start saving up for a LTO4 tape drive.
-----------------
Now for a filesystem for [any] drive, I'd recommend ZFS. In the newer releases among many features, you get snapshots and block level deduplication.
You're using new words that your grandparents didn't know but you don't see yourself as an uneducated English-mangler, you see yourself as 'correct'.
Your grandparents use words that their grandparents would have frowned upon.
Shakespeare is one of the greatest English writers ever but his words are incomprehensible to most people.
Your grandkids will use words that mean nothing to you.
It's called language...it'll change irregardless of what you think. Get over it.
No sig today...
This is the kind of topic that SHOULD be on slashdot. And no Im not being sarcastic. Something informational that might benefit everybody, dispite all the bickering in the subsequent posts! There's one thing you can count on with slashdot readers, dispite how arrogant most of us are, or how ignorant most are when it comes to politicial information - We usually know our shit when it comes to computer stuff! :)
My requirements weren't as stiff, but I found myself with an external drive that was used for offsite copying/sneakernet use between my *nix box, the wifes Mac, and friends Windows boxes.
I don't recall what FS was used, but it wasn't FAT or NTFS. I did set up a FAT partition with driver installer packages for that filesystem for most major OS's along with a README file with general instructions for mounting it on Windows and OS X and Ubuntu.
As long as there are tolerable drivers for any given OS, this gives the freedom to use the right filesystem for the task without sacrificing interoperability.
I myself wanted a setup like this several months ago and settled on setting up 2 Ubuntu systems with internal 2TB hard drive. One at home and one at my remote location. I share the drive over my local network with Samba. (The drives themselves are formatted ext3 or ext4, but that hardly matters to the rest of the network once they are shared.) (To later support Mac OS X you could try getting afp working, but it will still happily mount samba also, which is what I do with my mac.) Which is fast enough to serve video, and means I don't have to lug a drive around the house. And I periodically run Unison to keep the systems in sync over SSH. Even pre-mirroring the drives, Unison takes a day or so to run the initial sync. But after that first run, the syncs are pretty quick. At the moment I am manually running the GUI Unison, but I plan on eventually switching to the command line version and automating it in a cron task.
Today, after what -- 10 years being here? -- I got my first -1 for pointing NTFS in encumbered... and nobody cared.
That's normal everywhere... except here on /.
Today IMHO -- after a long descent -- /. went under the sea level.
Shall I perhaps register? (rhetorical)
I have been thinking about this recently and I had wondered if I could format a Harddrive as UDF like a big DVD. This format is readable by both Windows and Linux and I believe is writable to, although I have only ever used it on write-once media.
I got as far as finding Windows and Linux formatting commands as follows:
Windows Vista or later: "format x: /fs:UDF" (don't use /q ! ) /dev/sdx"
Linux: "mkudffs --media-type=hd
However, I have never actually tried it yet; so if anyone else has I would be curious as to how successful they were.
Western Digital apparently felt that well established power management standards were not hip enough and re-invented the wheel, only ribbed for their pleasure (not yours).
MyBook drives time out and go to sleep on non-windows systems every minute or so, and the linux and XNU kernels react to this rather poorly (in the case of linux kernels, by automagically remounting the drive RO which is actually worse than just erroring out).
They don't work right on windows systems either, unless you load the special drivers that slow down your whole OS.
Treat "MyBook" drives like the utter dogshit they are. That's how WD has been treating their customers who have complained, so it's appropriate. This post soon to be modded flamebait by WD staff....
Does anyone know if the other WD stuff is also non-standard crap? I have been afraid to buy anything from else them since they burned my ass on the 1TB MyBook.
http://www.google.com/search?q=western+digital+mybook+%22times+out%22
I'd choose the file system based on the system I plan to Host it on. If the host is Windows, I'd go with NTFS. If the Host is Linux, I'd go with EXT2. Regardless of the hosting system and the drive format, you want to make sure you can access the shared contents of the drive from your other systems on your home network. Using this approach, you won't need to worry about third-party drivers.
(OK, I did this for another drive that would have similar problems, I did this about 18 months ago IIRC)
Link talking about this with FreeAgent drives spinning down and giving I/O errors and remounting the drive read only. (Sounds very familiar doesn't it?) (I have a OneTouch drive which also suffered from the problem). This is how they got the drives working:
NSLU2 FAQ on FreeAgents spin down
(They do seem to make a mention that what they are writing to in the "/proc" tree might not have been there until sometime around 2.6.19)
FAT32: no POSIX metadata, 4GB file size limit is deadly. It's inefficient, and generally outdated and nasty.
NTFS: proprietary, sometimes complicated to get on Linux, hard to get on OSX.
ext*: bad-to-none support on non-Linux. IIRC, neither the Windows nor OSX drivers support journaling, for example.
HFS+: about the same boat as ext*, if you swap "Linux" and "OSX".
UDF: reasonably efficient, support for basic metadata (POSIX, though no EAs or forks), full support on Linux 2.6, OSX 10.5, Windows Vista/7, or (with third-party utils)
AFAIK, Windows 7 doesn't make NTFS partitions it comes across into Dynamic volumes (which is probably what you mean by "enhancing"), which would make them unreadable on Linux.
Maybe the directories you were trying to access were NTFS versions of symlinks, "junctions"? I've had trouble using them in Linux, and I know Win7 has them in place for backwards compatibility ('C:\Documents and Settings' points to 'C:\Users', for example).
WHO NEEDS SHIFT WHEN YOU HAVE CAPSLOCK/ DAMN1
>>WRONG!
I hate people that dont no how too use thee BIG letters and thee TINY letters rightly.
Wrong! Would be a more better way to right.
Thanks. Would mod interesting or insightful.
(founded 95,000,000 yrs ago, very space opera)
Anyone who uses "irregardless" is an idiot.
Used ext3, tried to install the drivers on my Mac and gave up. Came to a simple conclusion. If it's not Ubuntu, **** it.
Multiplatform. Decent. You may pick one. But now of course ext4 is out, which is much better than 3, so go with that instead.
It has better dynamic range than anything your digital harddrives can offer
ntfs-3g is fine for everyday use now. you can write and everything, with no silly limitations.
The only caveat that I know about is that occasionally (for example if the disk was not
cleanly unmounted) you might need to connect the disk to a windows pc once or twice to
let windows 'fix' it.
really, my brother and I have been using it for a couple of years now with no problems, and
it is the standard way of mounting ntfs partitions in any modern linux distro.
having said that, if your WD thing supports ext3, i would say try the 'ext2 ifs' driver for windows.
at http://www.fs-driver.org/
it is good enough for read access in windows.
and it works ok for writing though in my experience it occasionally refuses to delete a file.
well said randallman, lets not give even more power anyone who would even patent "that thing which i sometimes look out of". also, i have a related question about nas / fat referred to in this thread. i bought a nas drive, thinking that it would be a great backup solution. as far as i can ascertain i don't have an choice with the filesystem, only fat. as soon as i discovered this i decided nas wasn't for me and bought an eSata docking station which allows me to plug in any SATA, inc laptop, drive. i've not started playing, i.e. testing, it yet but i certainly hope i will be able to use any filesystem. i'm currently thinking along the lines always use EXT3, the eSATA is plugged into a Linux box so any windows machines could simply access it via the network. am i wrong?
Ubuntu 9.10-KK Kernel 2.6.31-16 - generic SMP x86_64; AMD PHENOM2 X4 945 sAM3 95W; GByte GA-MA785GMTUD2H AMD785G SB710;
+ 1 for build a NAS. Your backup doesn't need to be very portable. I started out with FreeNAS, if you go this route ZFS would be the logical choice. I got pretty frustrated with this so rebuilt my NAS using Ubuntu and EXT3 - it made sense to me to use the native FS for the OS. I currently have 4 x 1TB internal drives in raid 5 (one spare). I use a USB drive to back this up. SMB internally and all the files are accessible by every OS I use. If I need to take files out and about there are good old USB sticks or portable drives. Mine are usually NTFS or FAT 32 because anything can read these - and of course there's always good old FTP on the NAS