Slashdot Mirror


Best Filesystem For External Back-Up Drives?

rufey writes "I've recently embarked on a project to rip my DVD and CD collection to a pair of external USB drives. One drive will be used on a daily basis to access the rips of music and DVDs, as well as store backups of all of my other data. The second drive will be a copy of the first drive, to be synced up on a monthly basis and kept at a different location. The USB drives that I purchased for this are 1 TB in size and came pre-formatted with FAT32. While I can access this filesystem from all of my Windows and Linux machines, there are some limitations." Read on for the rest, and offer your advice on the best filesystem for this application. "Namely, the file size on a FAT32 filesystem is limited to 4GB (4GB less 1 byte to be technical). I have some files that are well over that size that I want to store, mostly raw DVD video. I'll primarily be using these drives on a Linux-based system, and initially, with a Western Digital Live TV media player. I can access a EXT3 filesystem from both of these, and I'm thinking about reformatting to EXT3. But on Windows, it requires a 3rd party driver to access the EXT3 filesystem. NTFS is an option, but the Linux kernel NTFS drivers (according to the kernel build documentation) only have limited NTFS write support, only being safe to overwrite existing files without changing the file size). The Linux-NTFS project may be able to mitigate my NTFS concerns for Linux, but I haven't had enough experience with it to feel comfortable. At some point I'd like whatever filesystem I use to be accessible to Apple's OS X. With those constraints in mind, which filesystem would be the best to use? I realize that there will always be some compatibility problems with whatever I end up with. But I'd like to minimize these issues by using a filesystem that has the best multi-OS support for both reading and writing, while at the same time supporting large files."

88 of 484 comments (clear)

  1. The solution.. by Anrego · · Score: 5, Interesting

    is to stop being so diverse! Pick a platform and stick with it!

    Ok, in all seriousness.. here's what you do:

    - buy yourself a cheap (~200) box
    - hook all your drives to it
    - use whatever file system you want (JFS, XFS would be my recommendation)
    - share it over your zoo of a network using nfs, samba, etc..

    As a bonus, your file server box could double as a media center, and replace your WD TV Live dealie.. (probably not though.. right)

    Irregardless, I think you're way better off with this approach vice trying to find the magical widely supported cross platform file system with large file capacity.

    You also might want to consider RAID vs. your monthly sync. Yes, RAID isn't a backup.. but for something like this where
    restoration would be possible, but just a pain in the ass.. mirrored raid can be a lot more convenient. You can always have
    a third external to back up your irreplacable data on a semi-annual basis..

    1. Re:The solution.. by Dynedain · · Score: 4, Informative

      Thats exactly what I did. Threw a couple of external drives on a Mac Mini. Formatted as HFS+ and did software array. Then using afp and smb provided the contents as shares to the Windows media center and various client machines on the network.

      Sure, software raid over USB is slow, but the bottleneck is the network so it doesn't really matter.

      --
      I'm out of my mind right now, but feel free to leave a message.....
    2. Re:The solution.. by Keruo · · Score: 3, Informative

      One shoud never consider raid vs synced copies, use both simultaneously. They protect against different data-loss threats which aren't mutually exclusive.

      --
      There are no atheists when recovering from tape backup.
    3. Re:The solution.. by rpresser · · Score: 5, Insightful

      It might if irregardless was actually a word.

    4. Re:The solution.. by uradu · · Score: 3, Interesting

      Or get a cheap NAS like the D-Link DNS-321. While certainly far from the bee's knees in terms of performance or number of bays (2), it can be had for under $100 and has been hacked to death to run all sorts of other stuff on it. Plus it's nice and quiet and doesn't use much power. And it's kinda purdy.

    5. Re:The solution.. by multimediavt · · Score: 5, Informative

      WRONG!

      Irregardless is not a proper English word. Its usage has *ALWAYS* irked me from when I was a small boy to now. To use common vernacular, it's a mashup of 'irrespective' (one negative; prefix) and regardless (also, one negative; suffix). 'Irregardless' is a double negative and is thusly illogical by construction and would only be understandable to people born in the U.S. since 1970, and those less literate in the U.S. prior to that.

      On my words that aren't words list it's right up there with 'impactful'.

    6. Re:The solution.. by David+Gerard · · Score: 3, Insightful

      +1 to this answer.

      This is what SAMBA is for. My home network has a mix of Ubuntu, Mac OS and Windows. It just serves to all of them without problems.

      I'm using a small silent PC as the server. Plug the USB drive into it, plug the USB turntable into it (and the cassette deck into the turntable) for ripping LPs and tapes, it's lovely.

      SMB over wifi serves fast enough to play MPEG4 video on the laptop and keep the toddler amused.

      --
      http://rocknerd.co.uk
    7. Re:The solution.. by uglyduckling · · Score: 4, Insightful

      Irregardless is nonsense caused by confusion between the words irrespective and regardless.

    8. Re:The solution.. by jeffehobbs · · Score: 2, Informative

      I second this (DNS-323 myself). Runs like a champ, very low-power, files
      available to every machine (and a WD TV Live) from any room in the house.

    9. Re:The solution.. by molarmass192 · · Score: 5, Funny

      You're words are truthy enough, but your assuming that synergistic words like irregardless don't have impacts on english as we know it. The facts is that people will use words like that wether we like it or not. This is truely, the case when it comes to American's use of language. Sadly, theirs very little we, as people far more litterate than the average people, can really do about that. If people used grammer checkers, then you and me would not see so many people authoring bad words and having a negative affect on english as it is known and practised today but should be editted and spokened tomorrow.

      --

      Good people do not need laws to tell them to act responsibly, while bad people will find a way around the laws-Plato
    10. Re:The solution.. by EdIII · · Score: 5, Insightful

      I've seen it used multiple times in this discussion. That makes it a word.

      ummmm. okay. "Fugnutish". Let's keep it going a few more replies and I gots my new word :)

    11. Re:The solution.. by Tanktalus · · Score: 5, Funny

      No, most people use it because they think they sound smart when they use a big word. The problem is, it's not a word and thus they just sound like an idiot to the very people they are trying to impress when they say or write irregardless.

      As a former coworker once told me, "Never use a large word when a diminutive one will suffice." I think he was showing off.

    12. Re:The solution.. by buchner.johannes · · Score: 3, Funny

      Irregardless, you're new word is stupid.

      --
      NB: The message above might reflect my opinion right now, but not necessarily tomorrow or next year.
    13. Re:The solution.. by camperdave · · Score: 5, Funny

      What's the difference between irregardless and regardless?

      Irregardless works in the dark.

      --
      When our name is on the back of your car, we're behind you all the way!
    14. Re:The solution.. by Jeff+Carr · · Score: 3, Insightful

      You can't just create words out of the blue. People will never be able to grok your meaning if you do. You know I'm right, so don't be all fugnutish about it.

      --
      The television will not be revolutionized.
    15. Re:The solution.. by RogL · · Score: 3, Funny

      I'm not sure I like the fugnutish tone you used in your post!

    16. Re:The solution.. by yukk · · Score: 5, Funny

      Utter rubbish. Irregardless is a perfectly cromulent word.

      Exactly. It embiggens the language.

      --
      The trouble with the rat race is that even if you win, you're still a rat." Lily Tomlin
    17. Re:The solution.. by Simmeh · · Score: 3, Funny

      He wasn't being fugnutish at all, don't you even know the definition of the word?

    18. Re:The solution.. by Sancho · · Score: 2, Informative

      100mbit link provides approximately 10MB/s. It's the difference between megabits and megabytes. I get about twice that on my USB drive. With gigabit, I can get around 50MB/s, and then the USB drive becomes the bottleneck.

    19. Re:The solution.. by MightyYar · · Score: 2, Insightful

      Irregardlessly, it is funny.

      --
      W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
    20. Re:The solution.. by Jah-Wren+Ryel · · Score: 2, Informative

      You can't just create words out of the blue. People will never be able to grok your meaning if you do. You know I'm right, so don't be all fugnutish about it.

      Don't be such an asshat.

      --
      When information is power, privacy is freedom.
    21. Re:The solution.. by zogger · · Score: 2, Insightful

      Those defortunate wooshers must have disirregarded all previal sarcatically full replitations yoused hear. I agreed, be-shortened bus haul of flaim.

    22. Re:The solution.. by socz · · Score: 2, Insightful

      You must be knew hear....

      :P

      --
      My abilities are only limited by my imagination
  2. Don't bother by Anonymous Coward · · Score: 2, Informative

    If you're like me, you won't be happy with the compromises you have to make when picking a multi-platform filesystem. I'd outline them, but you've done a great job of doing so above. So, what to do?

    Get thee a cheap, cheap Linux box, format your drives EXT3, and all other machines access over the network. It's the only way you'll get the interoperability you want, without making compromises on max file size, cluster sizes, etc.

    1. Re:Don't bother by tknd · · Score: 3, Informative

      If you're going to build a box, why not use FreeNAS with ZFS? It installed to a USB stick and everything is configurable from a web interface.

  3. NTFS by Hamsterdan · · Score: 4, Insightful

    THat's what I use here. for MAC, there's NTFS-3G (free) or Paragon ($ but faster on writes).

    I use NTFS on both my machines (Win/OS X/Linux) without any problems.

    NTFS-3G is also available for Linux.

    --
    I've got better things to do tonight than die.
    1. Re:NTFS by elh_inny · · Score: 2, Informative

      I second this choice.
      For some zealots it's hard to admit but the performance is really good, you have commercial backing of the biggest software company on the planet.

      Recently a commercial company (Tuxera) was formed to provide commercial support for NTFS-3G and provide paid-for version of the driver for MacOS and Linux in addition to the free NTFS-3G.

      So the future and cross platform access is looking really good.

      On the other hand, if I were just a little bit more adventurous, I would much rather use Sun ZFS for storage to have even better reliability, flexibility than with equivalent hardware RAID. But this pretty much requires a separate NAS box running OpenSolaris just for that.

      For me and many other garage hackers that just doesn't cut it, all I have is one laptop running a really fancy looking BSD and two external drives (NTFS) and some backup scattered over the Internet...

      All I'm hoping for is that one day Apple will reintegrate ZFS support, it's already been promised, implemented and now ditched...

    2. Re:NTFS by rhavenn · · Score: 2, Informative

      I second this choice.
      For some zealots it's hard to admit but the performance is really good, you have commercial backing of the biggest software company on the planet.

      You only have commercial backing if the OS you're running it on starts with Windows and doesn't include Linux, Solaris, BSD, AIX, Haiku, Amiga OS, etc... in the name.

    3. Re:NTFS by rpresser · · Score: 2, Informative

      UDF fits that bill, doesn't it?

    4. Re:NTFS by PhunkySchtuff · · Score: 2, Interesting

      As much as I love ZFS, one thing it's not good at is for a single removable disk. The inability for Apple to get this working without kernel panicking the machine was one of the many reasons they chose to drop it.

  4. Re:Ext3 by moderatorrater · · Score: 5, Informative

    I personally go the other way. Sure, there's an Ext3 driver for windows, but from what I've seen it's not that good. On the other hand, I've used the NTFS driver on Linux quite a bit and it's worked pretty well. More importantly, I have confidence that the NTFS driver will continue to get better.

  5. Fat32 and VLC by caubert · · Score: 3, Informative

    VLC can play rar-compressed-splitted files beautifully, So 4GB is not a very big problem

  6. The best is... by FreemanPatrickHenry · · Score: 5, Funny

    ...ReiserFS. I hear it's killer.

    --
    I have discovered a truly marvelous .sig which, unfortunately, this space is too small to contain.
    1. Re:The best is... by uglyduckling · · Score: 3, Funny

      Yeah, but it's murder getting support.

  7. I wouldn't.... by fak3r · · Score: 4, Interesting

    I wouldn't limit myself to a certain filesystem, I'd run a dedicated NAS like FreeNAS and share it over the network via SMB (windows), AFP (apple) and whatever for Linux - all set. Plus as mentioned above, you can run Firefly media server, a bittorrent server, a DAAP server (itunes sharing), etc (all included in FreeNAS. http://freenas.org/) on the same box. And since filesystems don't matter in this config, you can use ZFS to make a RAIDZ pool of your drives. It's what I do now.

    1. Re:I wouldn't.... by Barny · · Score: 3, Informative

      I used to use freeNAS, but after a while I just wanted more than what it was offering.

      I switched it for a windows home server (server 2003 SBE based), mainly for the backup features, and what with the freeNAS machine being the only non-windows machine left in my house it didn't matter that it lacks full compatibility with unix.

      But yes, freeNAS is damn good at what it does, have set up some nice diskless server systems with freeNAS running from a USB stick and having all the client machines on the network sharing their drives with iSCSI, freeNAS would collect them all, turn them into a big redundant storage array, and share them back to the network, works well :)

      --
      ...
      /me sighs
  8. ZFS, supported equally on your OSes by BitZtream · · Score: 3, Informative

    Via FUSE you'll get consistent features and useability across all 3 OSes. Of course moving zfs drives between those OSes isn't something I've tried, but in theory it should work fine.

    Not what your asking for, but Id put a FBSD samba server up with ZFS drives. You can still mount them on other OSes later if need be via FUSE.

    --
    Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
    1. Re:ZFS, supported equally on your OSes by Mr.+Protocol · · Score: 3, Informative

      Sadly, if you create a ZFS tank on a Solaris box and then move the tank physically to a FreeBSD 8.0-RELEASE machine, it won't even see that there's a tank out there. Apparently GPT table layout is different on FreeBSD or something.

      Won't stop you from serving ZFS over NFS/Samba/whatever, but you can't move the tank itself around. I know, I tried. Booted FreeBSD on a machine with a Solaris-issue ZFS tank, and it was like it wasn't there. It saw the drives fine, just not the tank.

    2. Re:ZFS, supported equally on your OSes by Mr.+Protocol · · Score: 3, Informative

      I should mention that the tank on disk was ZFS v4, so it was not a case of the Solaris tank being of a rev level higher than what FreeBSD could handle.

    3. Re:ZFS, supported equally on your OSes by joseph.spiros · · Score: 2, Informative

      zfs-fuse does not currently compile for platforms other than Linux, despite FUSE's general compatibility with other systems. The Mac OS X ZFS port worked for me under Leopard, and I think it works under Snow Leopard now, though it is no longer being developed by Apple. Last I checked, the ZFS pool version supported by these two implementations differs, so you'll need to create your pool with the latest commonly supported pool version to ensure compatibility. In short, I do support the use of ZFS, and I personally use zfs-fuse, but it's of far more use when shared over the network due to the problems with ZFS implementations on various platforms.

    4. Re:ZFS, supported equally on your OSes by zulux · · Score: 3, Interesting

      Did you "zpool export tank" before you moved the drives over? If not, then the FreeBSD box saw the drives but said to itself "Those drive don't belong to me!" I've done a migration from FreeBSD to Solaris, but always with full drive devices ie /dev/ad0 - thus ignoring partitions tables and other such cruddyness.

      --

      Moneyed corporations, non-working 'poor' and criminal prisoners are turning productive citizens into tax-slaves.

    5. Re:ZFS, supported equally on your OSes by g00ey · · Score: 2, Interesting

      But which is best platform to run, FreeBSD or OpenSolaris? It's difficult to find a good SAS/SATA controller that is well-supported by OpenSolaris. I have found a few cheap LSI MegaRAID conrtollers in China.

  9. NTFS is becoming the lingua franca by FreelanceWizard · · Score: 4, Informative

    Honestly, if FAT32 won't do what you need, NTFS is pretty much where you'll need to go. NTFS-3g gives you stable read/write capability on Linux and OS X as a FUSE driver; in fact, many distributions have NTFS-3g in their repositories. There's also native NTFS write support in Snow Leopard if you want to risk turning it on. I personally haven't had any issues with it, but some people have encountered file corruption when using it, so you might want to be wary. It is worth noting, however, that many embedded devices won't read anything other than FAT. If you plan on hooking this drive up to, say, a DVD player to show pictures, NTFS won't work for you.

    Like it or not, Microsoft file systems are the lingua franca of file transfer on portable drives these days, merely due to the installed base of Windows computers.

    --
    The Freelance Wizard
  10. Words of caution by RenHoek · · Score: 5, Informative

    I have ~6TB on external USB drives and I've been doing this for a few years now. I have a few words of caution about NTFS. If you get an USB drive that for example spins down or if you turn your USB drive off without properly dismounting it (or if Windows crashes), you might see this line:

    Delayed write failed!

    And on two occasions that meant that Windows fucked up the file allocation table or whatever it's called under NTFS and I lost the _entire_ disk.

    Windows loves getting its fingers into that table whenever you mount a USB filesystem. It's not like it tries to keep its write cache empty. Nooo.. every file access needs to be continuously recorded in that thing.

    Anyway, be careful when you use NTFS on a USB drive. Alternatively use EXT3, which you can still mount under Windows using:

    http://www.ext2fsd.com/

    (Note that these experiences are under Windows XP, I have no clue if Vista or 7 does any better, I assume not.)

    1. Re:Words of caution by Hamsterdan · · Score: 2

      Weird, because on both XP and 7 (on two different machines), the external USB drives are set for quick removal by default (meaning cache is disabled by default).

      --
      I've got better things to do tonight than die.
    2. Re:Words of caution by compro01 · · Score: 3, Informative

      Turn off write caching for the drive and this problem goes away. It's supposed to be off by default (at least on removable drives, but some IDE/SATA-to-USB bridges show up as normal fixed drives rather than removable for whatever reason), but I've found it seems to turn itself on for whatever stupid reason.

      --
      upon the advice of my lawyer, i have no sig at this time
    3. Re:Words of caution by spire3661 · · Score: 2

      How do i get external USB HDDs to allow the right click option to eject it, like i get from USB flash drives, on windows.

      --
      Good-bye
  11. network it by Sloppy · · Score: 3, Interesting

    I realize that there will always be some compatibility problems with whatever I end up with.

    Not if you use a network filesystem, such as Samba and NFS for the Windows and MacOS machines. Then on the Linux fileserver side, use whatever filesystem you want, and any OS can talk to that server.

    --
    As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
    1. Re:network it by zmollusc · · Score: 2, Interesting

      I have been using freenas on old boxes aggregating old disks ( I always want to get every last hour of use out of hardware before throwing it away) for a few years now and I find that the windows clients (98/2000/XP/Vista) work really well, much better than ubuntu samba. Thus i use nfs sharing for linux and smb for windows from the same freenas.
      I am rather crap at setting up samba and the freenas makes it all nice and simple for numpties like me.

      --
      They whose government reduces their essential liberties for temporary security, receive neither liberty nor security.
    2. Re:network it by StuartHankins · · Score: 2, Interesting

      And Windows clients frequently have compatibility issues with Samba servers all the time, especially when Microsoft releases updates to the client software. Esp. when it comes to things like domain membership, and file permission.

      Certainly not in my experience. Our main fileserver is RHEL 5.4 and we have approx 100 people over 19 locations connecting to it daily. It gets its data from our SAN. At 3 locations we have Fedora 8 systems serving up Samba shares (large local files that aren't good candidates for the WAN). The boxes are set to use domain credentials (Windows 2000 DC's) and everything passes through as you'd expect -- no prompting for credentials and new users' home folders are setup automatically the first time they connect. We also have mopiers in all 19 locations (some locations have several) accessing -- you guessed it -- the RHEL 5.4 fileserver over the WAN. No issues, files are owned by that user (except for folders with sticky groups). Security is a mix of both UNIX and Samba permissions.

      Main Terminal servers are 2003 R2 (and all those users access data from this fileserver), user desktops are XP (and my MacBook Pro running 10.4) along with a handful of Vista and Windows 7 systems in use by MIS. Works very well for us. We've been doing this since RHEL 3. It's actually kinda boring since it requires such little maintenance. Several of these servers cross-replicate via rsync every night so the backup copies are always on at least one other location's server (not directly accessible by end users).

      Yes, one of these servers uses NFS. I forget which one, since I set it up it's been uneventful. I remember there was a reason I wanted to use NFS but can't remember why, and I'm too lazy to login over the VPN to see which one. I do remember it's something critical we needed and NFS worked better.

  12. Re:Ext3 by xSauronx · · Score: 2, Informative

    yeah, i use linux and have my exteral drive formatted as NTFS

    honestly, i cant recall having a single write-related problem and i do raw DVD video often enough. linx NTFS support has been solid for me for well over a year now.

    --
    By and large, language is a tool for concealing the truth. -- George Carlin
  13. I have a few suggestions by mysidia · · Score: 2, Interesting

    If proprietary filesystems are on the table, how about VxFS ?

    Another possibility is to use FAT with cross-platform backup software. Maybe you don't need a filesystem at all: if this really is for backups... why not just create lots of extended partitions on the device and use TAR ?

    AKA tar cf /dev/sdbXX -V 'VOLUME_A' /backup

    That's crude and hard to keep organized, but also effective. Also, Some proprietary backup products that will work on a FAT filesystem, and not require large file support, even to backup large files

    Or utilize a tool such as WINRAR that allows you to "split" a RAR file across multiple archives in chunks of a certain size, then store these files on a FAT filesystem.

    FAT is the most cross-platform, oldest. But has known issues with fragmentation, and lack of journaling, effects reliability.

    You could divide your backup volume into 2 partitions: one DOS/FAT partition with the bootable image and files required to 'load a virtual machine' that can see the files on the other partition in the preferred data format such as ZFS or FFS (e.g. pre-allocated eager zeroed thick VMware VMDK with 'split into 2gb files' enabled).

    Then you just make sure the system you plug the drive into can boot a VM, with your "backup/file access environment"

  14. What about a backup server? by mlts · · Score: 3, Interesting

    As an alternative to an external disk that goes to multiple machines, this might cost some, but perhaps consider a backup server?

    The advantages to this setup:

    1: The server initiates the backups, and can warn you in case something can't be read.
    2: Most backup software stores snapshots, and some deal with the full/incremental/different cycle by using synthetic full backups. This makes restores to a certain point in time pretty easy.
    3: More sophisticated backup software allows you to transfer backup sets to another media. This way, you just plug in a drive, do a transfer, and you have an offsite archive.
    4: If one of the backup client machines gets hacked or malware installed, existing data stored on backup media cannot be altered.

    The disadvantages:

    1: You will need an active computer which is significantly more expensive than a hard disk.
    2: Amanda/Zmanda for open source, Retrospect, Backup Exec, for commercial. The software costs a hefty chunk of change.
    3: You have to make extremely sure that the backup server box is locked down tight. If someone compromises your backup server, they got data of every box you have. If you can, perhaps consider buying a router to put the backup server behind and only allowing the vital ports incoming.
    4: Backup servers should have some redundancy for stored data. Because there is so much data stored from multiple boxes, a failure of a drive hurts more than on a normal machine.
    5: Restoring a machine may vary in difficulty.

  15. Replace the WD TV thing by NitroWolf · · Score: 2, Interesting

    Replace the silly little WD TV Live media player with a mITX system that's about the same size. Install Linux and XBMC and be done with it. You'll have the best possible media player on the planet, as much storage space in any configuration you want and the ability to expand everything when the time comes. No hassle, you'll have constant online backups available and you'll have a killer always-on media center.

  16. Check Tuxera NTFS by replicant_deckard · · Score: 2, Informative

    If you need Linux/Mac/Windows interoperability then we recommend NTFS for both Linux and Mac users. Instead of the old NTFS kernel driver you may want to check our open source NTFS-3G. It has read/write, and tons of options:
    http://www.tuxera.com/community/ntfs-3g-advanced/

    If you need just high-performance NTFS read/write, this is our offering for Mac users:
    http://www.tuxera.com/products/tuxera-ntfs-for-mac/

    If you need high-performance for a commercial Linux application or device, you may want to check this:
    http://www.tuxera.com/products/tuxera-ntfs-commercial/performance/

    Regards,

    Mikko Välimäki
    CEO, Tuxera Ltd

    1. Re:Check Tuxera NTFS by hacker · · Score: 2, Interesting

      Bzzt... NTFS can't handle filenames that ext3, XFS and other Linux-based filesystems can handle. I went through this dance with my Drobo (incidentally, do not EVER buy a Drobo, not if you care about your data; it's dangerous to store data on that device)

      ext3 and the Windows-side e2fs-explorer style packages are fine, or use Samba/CIFS and serve it up that way. I use rsnapshot on Linux to back up my Linux and Windows machines to my NAS, which is ext3-formatted.

      NTFS is fine, if you're only ever backing up or storing data that can be created on Windows machines, but not if you want to store data from other machines (i.e. back up a Linux machine for example).

  17. Openfiler anyone? by lacourem · · Score: 2, Informative

    Personally I like Openfiler. It can be picky about the hardware though. With that said, the speed is great, and I can mount iscsi on linux and windows. Has been stable as hell for me to boot.

    --
    when logic fails, bullshit prevails
  18. ext3 by solid_liq · · Score: 2, Informative

    I go with ext3 for this personally. NTFS doesn't store *nix style filesystem permissions, and causes various other issues with you Linux systems. With ext3, you can store all your files with all of your permissions intact, the filesystem is mature and trustworthy, and you can still access all of the files from any operating system by simply connecting the drives to a dedicated fileserver machine (an older computer or small device works perfectly for this). Simply share your files via NFS, Samba and ftp (if you need ftp access for something like xmbc). Having a dedicated machine for this means you can also script your replication to the secondary drive, so that you only have to attach the drive for the mirroring process to take place.

    This is the solution I've been using for about four years now, and it works great for me.

  19. Re:Ext3 by IdleTime · · Score: 2, Informative

    I have a Seagate 1TB USB drive and I used it both under Windows and Linux, so the FS is NTFS:
    Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
    255 heads, 63 sectors/track, 121601 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Disk identifier: 0xa4b57300

    Device Boot Start End Blocks Id System
    /dev/sdb1 1 121601 976760001 7 HPFS/NTFS

    I currently run Linux exclusively and I'm going to buy the new 2TB drives and I will format it with btrfs since I will only be using it under Linux, but I have had no problems reading and writing to the drive under Ubuntu 9.10 nor under Windows XP.

    --
    If you mod me down, I *will* introduce you to my sister!
  20. I've been there - NTFS all the way by GoatSucker · · Score: 2, Interesting

    I have been through exactly the same thing, and NTFS is the way to go. My main machine is a Mac, but I format all my USB drives as NFTS (through NTFS-3G FUSE). Though this is slower than formating as FAT or HFS+, it is so much more portable, it wins out. Windows machines can read them. My WD TV box can read them. My Mac can read them. They can store more than 4G per file (indeed, I have all my DVD isos on them). Ok, they can't be read on other macs without installing NTFS-3G, but I've yet to need that. For me, I've never had any problem with NTFS-3G , it's extremely reliable.

  21. I basically have the same setup by Kappy · · Score: 2, Informative

    My backup has been worked good for the last few years. I had a large windows tower filled with hard drives...
    * One of my HDD was encrypted using Truecrypt.
    * A scheduled Acronis task would image my OS drive to the encrypted drive.
    * I have another encrypted HDD in one of these KINGWIN KF-91-BK trays (http://www.newegg.com/Product/Product.aspx?Item=N82E16817121172)
    * After Acronis was done, I would kick off a little batch file to mount the tray drive (which was encrypted) and run SyncBack Pro to mirror the encrypted source. Then dismount.

    I just bought a few extra trays. Each month I sync all the other HDD in the system (including the encrypted drive) for my off-site backup. One flaw in my system was if one of the other "data drives" crashed, I would loose at most a month of changes. They were just music/videos and didn't change very much. I accepted this fault.

    I out grew the tower and recently built a cheap unRaid box. Most if it should work the same, except now I have redundancy for all my drives in the case of a crash.

  22. Re:Ext3 by Sophira · · Score: 5, Informative

    Sure, there's an Ext3 driver for windows, but from what I've seen it's not that good.

    Which particular driver are you referring to? There are a few.

    Personally, I use Ext2 IFS in Windows (it works for Ext3 too) and it is, hands-down, the stablest and best Ext2/3 Windows driver I've used. Every other one I've tried would have stability issues; with IFS I don't have to worry. (There's been precisely *one* time in pretty much years that the driver crashed on me, and that's when I was doing something weird and stupid; I don't remember what. But more importantly, it didn't do anything bad to the filesystem in that crash.)

  23. NTFS-3G works fine by SlightOverdose · · Score: 2, Insightful

    NTFS-3G, which should come standard in most distros, should be able to read and write NTFS perfectly. It's considered very stable. That said, my personal solution to this problem was to use EXT2 and install EXT2IFS on my windows machines. I had a small FAT32 partition on the USB disk with the EXT2 driver installers for Windows and MacOS, so if I ever need to read it on another computer I don't have to download anything.

  24. ext3 by steveha · · Score: 2, Interesting

    I format my external USB drives to ext3. Most of my machines are Linux anyway, and I can always plug the USB drive into my storage server and backup over Samba to any kind of drive supported by the storage server.

    ext3 is pretty much stable and well understood. It just works. That's what I want for backup drives.

    And my netbook has Ubuntu Linux on it, and ext3 performs well on the external USB drive there. I haven't tested NTFS over FUSE on the netbook, but I wonder about CPU overhead on the little Atom chip: it might be a little bit slow.

    If you want a drive you can take over to your friend's house, and your friend just runs Windows or a Mac, then by all means NTFS.

    steveha

    --
    lf(1): it's like ls(1) but sorts filenames by extension, tersely
  25. PSA by dotgain · · Score: 2, Informative

    Ignore this thread. Once you get past all the first post trolls, the rest of it mainly contains bickering over whether "irregardless" is a word or not. Slashdot at its finest, I know. So I'm jumping in at the top of the thread to warn you and save you the time.

    As for which filesystem to use: It's a shitfight at the end of the day. OSS-NTFS is reverse engineered so you can never trust that it really is complete, nor that Microsoft will never bend/break it in the future. FAT32 sucks but is unquestionably the lowest common denominator. I like to use Ext2 for getting big files from Linux to Windows (I transfer a lot of DV files), but you have to install something on the Windows/Mac/etc. to use that. HFSplus support on Linux is fine, you just need to disable the journal on the Mac first.

    Like I said, it's a shitfight. Learn the pros and cons of the main filesystems: NTFS, Ext-2/3 family, HFS, FAT32. Break them out as each situation merits. But with regard to NTFS and the specs changing: Microsoft have done it before and they'll do it again, just ask Norton Utilities. While my Windows PC is a very important part of my operation (I edit all my video on it), I don't use Windows for enough other things to justify using NTFS unless I'm forced to.

  26. turn your usb drive into a NAS by stilldead · · Score: 3, Interesting

    I haven't tried it but it looks like a good idea. http://www.cyberguys.com/product-details/?productid=36218&sk=MC71419
    Format it ext3 and then share it SMB for any OS.

    --
    You are lucky, Ed Gruberman. Few novices experience so much of Ti Kwan Leep so soon.
  27. Why all those ext3 recommendations? by the_other_chewey · · Score: 5, Interesting

    I see quite many people here recommending ext3. Oh my. ext3 sucks for large files,
    which is exactly what the submitter wants to use his setup for. Look into the crazy structures
    ("double indirect blocks") it uses. He should go with an FS that has sane data structures with
    files >>4GB.

    That kills most of the choices and leaves XFS, ext4, ZFS (only worth it if not used via FUSE,
    i.e. in Solaris), and a couple more obscure ones.
    I second the "forget OS portability, use a server" suggestion. There's great low-power, low-cost
    hardware for this nowadays.

    1. Re:Why all those ext3 recommendations? by Anonymous Coward · · Score: 2, Interesting

      I see quite many people here recommending ext3. Oh my. ext3 sucks for large files,

      The reason so many people are recommending it is because it's the best choice, not the fastest choice.

      XFS isn't popular enough. Yeah, it sucks, but unless there're other compelling advantages, I want to use code that has lots of other users to get rid of obscure bugs, especially for filesystem code.

      ext4 is too new and doesn't offer sufficient advantages over ext3 in this use-case.

      ZFS - Pretty much requires Solaris or FreeBSD. The poster obviously already knows Linux, whereas their Solaris/FreeBSD experience is an unknown quantity. Also, for the "generic" slashdot readers out there, Ubuntu/Fedora are a lot easier to install/setup than Solaris/FreeBSD (or Debian for that matter). Now, if ZFS was available natively under Linux, its snapshot/checksum/disk-pool features (all of which are so extremely cool and useful for a home-NAS) might provide sufficient advantage to overcome the fact it's less popular.

      ext3 is a mature filesystem that can store large files (large enough for home use anyway) and is easily fast enough to flood a gigabit network connection. It's not that the other filesystems sucks, it's just that ext3 fulfills the role quite adequately and the minor advantages of the other filesystems isn't sufficient to overcome its advantage as the "de-facto" filesystem.

  28. Re:Ext3 by randallman · · Score: 3, Insightful

    NTFS is recommended several times in response to this article and it troubles me. The next de facto portable filesystem standard is being determined by us in the same way mp3 was made a standard. Ask yourself if you really want to see NTFS become the standard? Personally I'd rather see a truly open filesystem become standard and I really don't want to see Microsoft have the leverage to make more patent threats. If you agree with me and have some integrity, please recommend an open standard.

  29. NTFS and Compress by ccrasher · · Score: 2, Interesting

    I would recommend NTFS. I would also recommend that you get an open source program and compress the dvd's to approx. 700MB for up to 90 minutes and 1.4GB over 90 mins. H264/ac3 or Xvid are good codecs to use. Batch schedule the movies and let the system compress all night when you don't need the system. The reason for this is because dvd's take up a lot of wasted space because of the format. A compressed movie takes up far less space. I can store about 1000 movies on a 1TB drive and if a dvd is ever needed, I can recode a dvd. There is not much quality difference between a compressed movie and a dvd file. You can also convert the dvd's to .mov files but apple can read most H264 files or mp4 files. Many dvd players can read Xvid or Dvix files and others without the need for a dvd file. For cd's, If you plan to burn several copies, then rip the cd's directly to .wav or .flac (lossless) files. mp3 files are like .jpg image files such that they lose a little quality/data each time they are copied whereas lossless .wav or .flac files DO NOT regardless of how many times they are copied.

    1. Re:NTFS and Compress by MightyYar · · Score: 3, Insightful

      I would recommend NTFS.

      That is the best solution to read from, but he says that he's going to primarily be using this from Linux - so NTFS might still be a bit risky.

      I would also recommend that you get an open source program and compress the dvd's to approx. 700MB for up to 90 minutes and 1.4GB over 90 mins. H264/ac3 or Xvid are good codecs to use.

      A couple of points here. First, why would he specify a file size unless he was going to put the movies on CDs? He'd be far better off simply using a constant-quality encoder, which would have the additional benefit of not requiring a second or third pass for optimal quality. Second, why recompress in this day and age? A 1TB drive can hold over 100 double-layer DVD movies... and if he has more DVDs than that, then the cost of a bigger/additional drive is minor compared to the cost of the collection...

      mp3 files are like .jpg image files such that they lose a little quality/data each time they are copied

      This is simply not true. Neither mp3s nor jpegs lose quality or data when copied. They only lose quality when they are re-encoded - just as you are proposing he do with his DVDs. DVDs are heavily compressed using MPEG2. Converting to MPEG4 or H264 is exactly the same as taking an MP3 and converting it to AAC or OGG - there will be quality loss no matter how well the encode is done.

      --
      W..w..W - Willy Waterloo washes Warren Wiggins who is washing Waldo Woo.
  30. Please read this by Anonymous Coward · · Score: 5, Informative

    Dear AP,

    I was pondering with a very similiar problem two years ago, when I bought my first terabyte-class external USB hard drive to my home server (old 800 MHz Duron). I was thinking exactly like you are thinking now, and decided FAT-32 would be the way to go. MISTAKE. Four important things stand out, that I want you to read:

    1. You never really transfer that drive. Much less than you think. Chances are more than 99% of their lifetime they will be sitting hooked to one computer, never really being moved; portability is just another nice extra feature that you hardly ever use. During the two years I have switched my drives a couple of times between the desktop machine and the server, both of which run Linux as main OS. I have never ever taken any of these drives outside of my home.
    2. 4 GB limitation is really bad! Most DVD ISOs are bigger that that. An hour of HD video in state-of-the-art H.264 is more than 4 GB. And rest assured, when you have the space and facilities to accquire gigabyte-class multimedia, the temptation will be there. As BluRay becomes the new DVD, maybe you want to RIP your fav. movies to your hard drive for quick access? NO WAY with FAT-32!
    3. Lack of UNIX parameter support. Okay, so you just want to store back-ups? Okay.. Remember than FAT-32 doesn't support symbolic links, file ownership, user/group/others access permissions, file name character case (in Microsoft Windows, "Soviet Union" equals to "soviet union"; WILL result in a conflict when copying data from UNIX filesystems!). This information is LOST, unless you use some container format like tar (but remember the 4GB limit again). These little things are a) very helpful everyday things, value of which you realize only after loosing them (e.g. any file on extfs can be replaced/virtualized without moving files around; it can even point to a non-existstent file! And all works seamlessly, as long as the program understands symbolic links; now how valuable is that?), and b) what makes your UNIX fs work. The value of your backups is lower if they dont work "out of the box", e.g. data is lost when transfering to FAT-32. I mean, you just have a chance to save so much hassle there.. When needed, you can NFS-mount the filesystem (and its free space and contents) to your local machine from your drive, and everything works transparently to BOTH Windows and Linux (the properties of FAT-32 are a small subset of those of extfs.)
    4. Acccess speed. Ext3 and Ext4 or just about any 21st century UNIX fs are lightyears ahead of the archaic FAT in data structures. E.g. if I "ls" a big directory on my only FAT-32 drive, it is SLOW! You can see the entries being fetched one by one. Whereas, if I do the same in a similiarly-sized directory on the ext3, the files appear immediately! Access is almous immediate even over NFS mount in LAN. This comes handy, rest assured.

    Okay, those are my four vocal points. They could be in any order, because all of them are equally important reasons NOT to choose FAT-32! As it happened, after using the 0.5TB drive for 6 months with FAT-32, I bought more space (a new drive). This time there was no question about the filesystem. I made a small, few-sector long 200 MB FAT-32 partition to the beginning of the drive and downloaded all the latest Win32 EXT2 drivers there from different vendors, just for the really unplausible situation that I would ever want to mount these drives in Windows. Then I just made the rest Ext3. And, I am REALLY satisfied with the decision! Ext3 just work so sparklingly faster and better with Linux than FAT-32 ever does. Since then I have bought one more drive and did the same 200MB + 1TB thing. I will probably never use these drives in Windows, but it gives me a warm feeling to the heart that there's always a way if I should, even if the computer doesn't have an Internet connection.

    Oh, one thing I forgot to mention: get a file server! It makes your life so much easier. Nowadays I am running a desktop computer with a 60 GB SSD drive and no HDD at a

  31. Problem with ReiserFS by TheLink · · Score: 5, Funny

    The big problem with ReiserFS: Vendor lock-in.

    --
  32. Re:Ext3 by creepynut · · Score: 3, Informative

    Except that it's basically unusable now.
    I don't know the details 100%, but the Ext2 IFS driver doesn't support Inodes that are larger than 128 bytes, according to the site you linked.

    My understanding is that Ext3 defaults to 256 byte Inodes now. The only way to change it is to manually create the file system with some flags, no gui option, certainly not while I'm installing Fedora or Ubuntu.

    I tried getting it to go but when I attempt to access the drives windows tells me it needs to be formatted before use. Not good!

    Perhaps it will eventually be fixed, but I will agree with the GP, using NTFS from Linux has been rock solid for at least a couple years for me.

  33. Oblig: XFS plug - it was made for media streaming by lpq · · Score: 2, Interesting

    XFS was designed with media streaming in mind -- and designed for large file, high performance. It had a defragmenter to keep disks in optimal condition before Windows98 had come out (was one at the request of a large, customer who had an especially pathological case -- before that there was normally not considered a need for it).

    Files can be 'normal', have up to and addition 256K of resource-fork related into (extended attribute info), AND you can have a real-time section that can allow for completely bypassing the file system. It was sufficiently fast for video even back when disks were 1/10th the speed they are now.

    On it's native OS, it could handle multiple streamed data to the same disk and keep it separate by allocating the separate channels out of disparate allocation groups on disk. I don't know how that works on linux. Unfornately, on linux even under x64, file block sizes AFAIK, are still limited to 4K. XFS has a 64K limit, but under linux is hamstrung to 4k. Of course Windows NT allows 64K block sizes. But not linux...hmmm....very weird. XFS minimizes impact of linux's tiny allocation block size by using a extents which can be at least 256k -- but believe the actual limit is in megabytes. Been a while since I read that stuff...

    Of course -- not to be linux centric, but have heard ZFS is pretty good, but no idea of how it compares for anything.

    my 2 cents...

  34. Avoid USB attached storage. (Re:Words of caution) by KonoWatakushi · · Score: 4, Informative

    The fundamental problem lies in USB bridge chips which do not properly implement the cache management commands. Others have replied that you need to disable the write cache, and while that would be a solution, it is often impossible. Even with bridge chips that do support the cache disable command, some hard drives will not honor it anyway.

    Most USB bridges simply lie about when data has been written, which makes it very difficult for a filesystem on top of it to make any guarantees. While it may not happen often, this can have disastrous results, as you have seen.

    The copy on write nature of ZFS left it especially vulnerable to broken USB storage, and could easily leave you with a corrupted pool requiring manual intervention and a bit of luck to recover. Thankfully, the recent bits address this, and ZFS is now the only filesystem that I would trust on top of USB storage. Most other filesystems will survive without incident, but at the cost of some silent data corruption.

  35. Hybrid? FAT32 and [your-choice-of-fs-here] by corychristison · · Score: 4, Interesting

    I know this comment will get lost in the sea of other comments, but my recommendation to you would be a hybrid solution.

    Create a small partition (1GB would be overkill) and format it FAT32.
    Create another partition for the rest of the drive (or however you please) with your choice of FS (I prefer XFS, personally).

    Store the drivers(/utilties) for the FS you chose and store them on the FAT32 drive.
    Some popular drivers/utilties for Windows are:
    ext2fsd for EXT2 - http://sourceforge.net/projects/ext2fsd/
    rfstool for ReiserFS - http://freshmeat.net/projects/rfstool/
    ltools for EXT2/EXT2/ReiserFS - http://www2.hs-esslingen.de/~zimmerma/software/ltools.html/
    and so on and so forth (a simple google for "[FS] Windows Compatibility" usually works.)

    Just my thoughts. :-)

  36. btrfs & data checksums by zerothink · · Score: 2, Interesting

    Great feature would be data chacksums - for media collection on drive, I would like to get warning if there is any data loss. btrfs has this feature (between many others), when it is a bit more stable, than I will use it in similar scanario.

  37. Bubba! by talornin · · Score: 2, Interesting

    I ended up doing as most in this thread did. Networking.

    I bought a BubbaII from http://www.excito.com/ its a small fan less linux box with 2x usb, 2x ethernet, and 2x extSata.

    NB: NICs are gigEthernet, but they perform substantially slower than one expects. This according to the manufacturer, is by design to keen the temperature at a resonable level to accommodate the fan less design.

    --
    When in danger, whewn in doubt! Run in circles, scream and shout!
  38. Failure on large files? by cheros · · Score: 2, Interesting

    I've been using this for years, but I do get solid lockups with very large files (over 4GB) which has made me very nervous.. Any idea where I screw up?

    Desktop and laptop are still WinXP (home) until I make up my mind what I'll replace it with (it's narrowed down to OpenSuSE or Mandriva), but for some use a Windows desktop is still more suitable (I'd love Paint.NET to run on Linux, for instance).

    --
    Insert .sig here. Send no money now. Owner may sue, contents will settle. Batteries not included.
  39. Create different partitions on external drive by MirzaD · · Score: 2, Informative

    Create different partitions on external drive, one for example EXT3 and other FAT32. EXT3 would be large and FAT32 would be very small couple of megabytes. On smaller partition you would have EXT3 drivers for Windows machines. With this you would be able to access EXT3 from any machine. You could also try different file systems in same way.

  40. Re:Ext3 by osu-neko · · Score: 3, Insightful

    NTFS is recommended several times in response to this article and it troubles me. The next de facto portable filesystem standard is being determined by us in the same way mp3 was made a standard. Ask yourself if you really want to see NTFS become the standard?

    No, but that's not really relevant.

    Personally I'd rather see a truly open filesystem become standard and I really don't want to see Microsoft have the leverage to make more patent threats. If you agree with me and have some integrity, please recommend an open standard.

    Can't reconcile the contradiction in that request. There's no open standard that satisfies the need, so I would have to abandon my integrity to make a dishonest recommendation that I like for its openness. If I want to maintain my integrity, I need to make an honest recommendation that accords with the truth, even if it's an unpleasant truth. For the desired purpose, the sad truth is your best bet is NTFS. I wish that weren't so, I would hope that in a few years some open standard would emerge, but I'd be dishonest to answer with wishful thinking rather than fact.

    --
    "Convictions are more dangerous enemies of truth than lies."
  41. nice... by ZenDragon · · Score: 2, Insightful

    This is the kind of topic that SHOULD be on slashdot. And no Im not being sarcastic. Something informational that might benefit everybody, dispite all the bickering in the subsequent posts! There's one thing you can count on with slashdot readers, dispite how arrogant most of us are, or how ignorant most are when it comes to politicial information - We usually know our shit when it comes to computer stuff! :)

  42. Samba and Unison by poolecl · · Score: 2, Informative

    I myself wanted a setup like this several months ago and settled on setting up 2 Ubuntu systems with internal 2TB hard drive. One at home and one at my remote location. I share the drive over my local network with Samba. (The drives themselves are formatted ext3 or ext4, but that hardly matters to the rest of the network once they are shared.) (To later support Mac OS X you could try getting afp working, but it will still happily mount samba also, which is what I do with my mac.) Which is fast enough to serve video, and means I don't have to lug a drive around the house. And I periodically run Unison to keep the systems in sync over SSH. Even pre-mirroring the drives, Unison takes a day or so to run the initial sync. But after that first run, the syncs are pretty quick. At the moment I am manually running the GUI Unison, but I plan on eventually switching to the command line version and automating it in a cron task.

  43. Which File System? by olsonde · · Score: 2, Interesting

    I'd choose the file system based on the system I plan to Host it on. If the host is Windows, I'd go with NTFS. If the Host is Linux, I'd go with EXT2. Regardless of the hosting system and the drive format, you want to make sure you can access the shared contents of the drive from your other systems on your home network. Using this approach, you won't need to worry about third-party drivers.

  44. Re:NTFS linux driver will always be quirky by yoyhed · · Score: 2, Insightful

    AFAIK, Windows 7 doesn't make NTFS partitions it comes across into Dynamic volumes (which is probably what you mean by "enhancing"), which would make them unreadable on Linux.

    Maybe the directories you were trying to access were NTFS versions of symlinks, "junctions"? I've had trouble using them in Linux, and I know Win7 has them in place for backwards compatibility ('C:\Documents and Settings' points to 'C:\Users', for example).

    --
    WHO NEEDS SHIFT WHEN YOU HAVE CAPSLOCK/ DAMN1
  45. Re:Ext3 by ppc_digger · · Score: 2, Informative

    It has at least two other issues: it doesn't support ext3 journaling (so ext3 is treated as ext2), and it can't fsck (and because it doesn't support ext3, it can't do a journal replay either), which means that it can't mount a dirty volume until it is mounted by Linux.

    --
    Of all major operating systems, UNIX is the only one originally meant for gaming.
  46. NAS is the way to go by seangee · · Score: 3, Informative

    + 1 for build a NAS. Your backup doesn't need to be very portable. I started out with FreeNAS, if you go this route ZFS would be the logical choice. I got pretty frustrated with this so rebuilt my NAS using Ubuntu and EXT3 - it made sense to me to use the native FS for the OS. I currently have 4 x 1TB internal drives in raid 5 (one spare). I use a USB drive to back this up. SMB internally and all the files are accessible by every OS I use. If I need to take files out and about there are good old USB sticks or portable drives. Mine are usually NTFS or FAT 32 because anything can read these - and of course there's always good old FTP on the NAS