Best Format For OS X and Linux HDD?

dogmatixpsych writes "I work in a neuroimaging laboratory. We mainly use OS X but we have computers running Linux and we have colleagues using Linux. Some of the work we do with Magnetic Resonance Images produces files that are upwards of 80GB. Due to HIPAA constraints, IT differences between departments, and the size of files we create, storage on local and portable media is the best option for transporting images between laboratories. What disk file system do Slashdot readers recommend for our external HDDs so that we can readily read and write to them using OS X and Linux? My default is to use HFS+ without journaling but I'm looking to see if there are better suggestions that are reliable, fast, and allow read/write access in OS X and Linux."

Comments

UFS.

[necroplasm]

UFS would be the best option. Linux supports it with -rw since Kernel 2.6.30 (afaik) and OS X mounts UFS natively.

Re:UFS.

[clang_jangle]

UFS would be the best option.

Unless you're using Tiger or earlier, UFS is not an option. The last two versions do not support UFS at all. However, HFS+ support in Linux is pretty good. Otherwise you're looking at mac-fuse for ext2/3, which IME is pretty slow and buggy. I thinks Jobs has gone out of his way to make OS X incompatible with OSes other than windows. Maybe he's afraid of what will happen if everyone becomes aware they have other choices.

Re:UFS.

[X0563511]

Every filesystem warrants the occasional check. If you never check, there are lots of errors that can accumulate and burn your ass.

Re:UFS.

[RobertM1968]

...storage on local and portable media is the best option for transporting images between laboratories. What disk file system do Slashdot readers recommend?

Every filesystem warrants the occasional check. If you never check, there are lots of errors that can accumulate and burn your ass.

Methinks you may be plugging your portable media into the wrong place... then again, I've never tried that, so I could be wrong. ;-)

Re:UFS.

[dogmatixpsych]

Yes, the raw files from the scanner are quite small. A whole series of scans (7 or 8 high quality sequences) is only about 450MB. We get 80GB files when we do post-processing (fiber tracking) of a diffusion scan.

Re:UFS.

[Anonymous Coward]

It's the default filesystem in *BSD, so it's very well maintained etc. It has journalling (or does it call it "soft updates"?) auto-defrag, etc, etc. You fsck it if you power off without umount but otherwise you won't need to.

It's definitely a perfectly capable, full-featured, modern filesystem.

All the things you write are perfectly true... on *BSD variants where UFS is the native, default FS. That is not the case on either Linux or OS X, to the extent that in OS X v10.6 UFS is now a read-only FS because it's barely maintained.

Most people who think OS X is truly 'native' on UFS because it has BSD heritage haven't tried to actually use it. When Apple bought NeXT in 1997 the UFS implementation was already behind the times because at that time NeXT hadn't been updating its operating system for a few years. Since Apple wanted OS X to be a MacOS upgrade, development resources went into making a robust and high performance HFS+ implementation. Very little was done to modernize UFS. From the outside, it seems to have been just enough effort to make sure it worked and was still bootable over the first few versions, for those who wanted native UNIX FS semantics (mostly case sensitive file names). Then they added case sensitive filename support to HFS+ (it's a format-time option), and since then there has been even less reason for Apple to maintain UFS, hence its transition to a read-only legacy format.

The other piece of this picture is that UFS != UFS. The UFS in MacOS X is a mildly upgraded version of mid-1990s NeXT UFS (which, in fine BSD tradition, wasn't quite the same as the UFS found in other BSDs). It's almost certain it has few of the features you associate with modern versions of UFS.

Re:UFS.

[dotgain]

You can get R/W support but only if you disable the journal on the HFS+ fs on MacOS. That's unless they've taken that away too in 10.6, but I routinely do this with HFS+ between 10.5 and Linux.

Followup question...

[serviscope_minor]

I have a similar problem, albeit on a smaller scale. I use unjournalled HFS+.

However, the problem is that HFS+ being a proper unix filesystem remembers UIDs and GIDs which are usually inappropriate when the disk is moved.

Is there any good way to get Linux to mount the filesystem and give every file the same UID and GID, like for non unix filesystems?

Re:Followup question...

[SEAL]

Many filesystems support uid= and gid= options in their mount command (including HFS). Just add that to a mount script or set it up in fstab.

Re:Followup question...

[X0563511]

Non-native filesystems usually let you set UID, GID, and permission masks. Check the "mount" manpage and look for the filesystem you want. You might also try "man filesystem"

HIPAA Constraints?

[fm6]

By "HIPAA Constraints" I assume you mean the privacy rule. I would think that this rule would prevent you from using sneakernet to transmit files. Unless you're encrypting your portable disks, and somehow it doesn't sound like you are.

Fun reading:

http://www.computerworld.com/s/article/9141172/Health_Net_says_1.5M_medical_records_lost_in_data_breach

Re:HIPAA Constraints?

[eschasi]

HIPPA mandates who can and should have access to the files. The method of storage (disk, tape, SSD, paper, whatever) is largely irrelevant. As long as all those who have access to the files are HIPPA-trained and following the appropriate procedures, everything is fine. Similarly, transport is relevant only in that there must be no data disclosure to unauthorized persons. As such, if a person with appropriate clearance does the transport, all is cool.

HIPPA data is often encrypted when placed on tape or transported across systems, but that's because such activities may involve the data being visible to unauthorized people. As examples of each:

• If two physically separate sites exchange HIPPA data across the open Internet, the data must be encrypted during transport. This might be done by VPN, sftp, whatever. As long as the bits on the wire can't be read by the ISPs managing the connection, it's OK.
• For tapes that you archive off-site, you don't want your external storage facility to be able to read the tapes, nor have the data readable if the tape is misplaced in transport.

IMHO wise use of sensitive data on laptops requires encryption at the filesystem level. It's neither difficult or time-consuming, but given how much sensitive data has been exposed via folks losing or misusing laptops, it ought to be a no-brainer. Sadly, too few places bother.

Re:HIPAA Constraints?

[rwa2]

Maybe instead of using a portable disk, they could whip up a nettop running Linux and transfer files over the gigabit ethernet...
Then they could do transfers via samba or rsync+ssh , and the nettop could transparently take care of encrypting the underlying FS, whatever that may be.

Performance wouldn't be great... maybe 20MB/s instead of 60MB/s for an eSATA drive, and they'd have to work out a consistent network port / IP across all the sites it travels to. But it might confer some advantages.

Along similar lines, they could put in a small file server at each site, and rotate a removable disk drive between all of the file servers. That way they'd just have a drop box that they could always push files to throughout the day, and let the couriers just grab what's there and deliver.

Re:HIPAA Constraints?

[dogmatixpsych]

All of our scans are natively anonymized: we make up the birthdate and we never include the research participants' names. Our images are high enough quality that you can do nice 3D reconstructions of people's heads (and faces) but there is virtually no chance that anyone would recognize the faces unless they knew a person really well (even then it is hard). We have checked with the privacy office and our drives do not have to be encrypted because the images that will be on these drives are de-identified. We probably will encrypt the drives but wanted to nail down a filesystem first before we deal with encryption.

Anyway, we only rarely will need to use the sneakernet but need the option. HIPAA is only a minor issue. The biggest issue comes in dealing with multiple IT departments and setting up network access to our materials. Plus our images are so large that for these processed files (not the originals) we are opting for local storage instead of storage managed by our IT staff (who are wonderful but not cheap; we just purchased 4TB of local storage for 1/4 the cost of 1TB from IT).

Re:HIPAA Constraints?

[rwa2]

Yeah, then it sounds like you're pretty much doing the best you can under the circumstances... I was just trying to think out of the box a bit and turn your filesystem compatibility problem into a file server compatibility problem, since cross-platform compatibility is a much bigger deal in the latter scenario.

One last consideration you might want to try benchmarking is storing your data in an image file, like a zip or tgz or more likely a dmg archive... that way you could probably do transparent compression as well, which might work well on your datasets. If your disks have limited performance from USB or whatever and you have beefy CPUs, that may even increase your overall throughput when making copies to/from fast local storage. Plus some of the archival formats may help preserve some filesystem features / metadata that may otherwise get clobbered by copying through a compatible intermediary FS.

Don't know if broke

[tepples]

More like "I don't know if it's broke. I could be doing something so wrong it could end up on The Daily WTF. If what I'm doing is broke, could you help me fix it?"

Re:Ugly but works

[GooDieZ]

and 4Gb cap...

FAT32 is a fucking horrible idea in his case.

[Anonymous Coward]

How the fuck is he supposed to store 80 GB files on a filesystem that maxes out at 4 GB?

Re:FAT32 is a fucking horrible idea in his case.

[Tirs]

Sorry man, but your use of the "f" word is totally inadequate in this conversation. Let me correct you:

How the fsck is he supposed to store 80 GB files on a filesystem that maxes out at 4 GB?

Much more in-context, eh?

NTFS

[Trevelyan]

NTFS or any other FUSE (MacFUSE) file system. However in a heterogeneous environment NTFS has the bonus of native Windows support.

There is NTFS-3G for Linux and Mac OS X

There is also an EXT2 Fuse FS (for Mac OS), and probably many other options.

Having said that, I have never had a problem with Linux's HFS+ write support.

Reiser?

[Wowsers]

I would have recommended ReiserFS, but the data might get buried somewhere and the system would not remember where it was....

Re:Reiser?

[Anonymous Coward]

That's pure pure FUD. ReiserFS can recover anything, even something it allegedly never stored. "Oakland homicide detective Lt. Ersie Joyner recalled that Reiser led them directly to the exact site, without any hesitation or confusion."

No Filesystem

[Rantastic]

If you are only moving files from one system to another, and do not need to edit them on the portable drives, skip the filesystem and just use tar. Tar will happily write to and read from raw block devices... In fact, that is exactly what it was designed to do. A side benefit of this approach is that you won't lose any drive capacity to filesystem overhead.

Re:No Filesystem

[Rantastic]

With that said, tar is a bad solution because it doesn't include any type of CRC or encryption. But it's a good idea, and certainly a million times better than a file system of some type.

True, but simply hashing the file at both ends solves that. Both linux and mac support shasum.

Re:No Filesystem

[Rantastic]

As to encryption, you just encrypt the file before you tar it. In fact, with gpg you get both encryption and integrity checking.

Gnupg is available in Mac Ports and comes with just about every linux distro.

Rubbish

[Improv]

You're storing it in the wrong format - there are all sorts of tools to convert to Analyse or DICOM format, which give you a managable frame-by-frame set of images rather than one huge one. Most tools to manipulate MRI data expect DICOM or Analyse anyhow (BrainVoyager, NISTools, etc).

If you really want to keep it all safe, use tarfiles to hold structured data, although if you do that you've made it big again.

Removable media are a daft long-term storage - use ad-hoc removable media solutions (or more ideally, scp) to move the data.

Re:NTFS

[X0563511]

Windows doesn't play in here, it's OSX and Linux. Tossing NTFS into that would just be... wrong somehow.

Re:NTFS

[Tirs]

Hah! In my company we call it "NoTeFíeS" (for non Spanish-speaking people: "Don'tTrustIt").

NFS over SSH

[HockeyPuck]

Just tunnel NFS over SSH. I can't imagine how secure it would be to sneakernet any files around the office. If you need to encrypt the data at rest then either encrypt on the client or leverage an encrypted filesystem of a Decru type appliance.

4GB per file limit

[Ilgaz]

OS X UFS has a very unfortunate limit as it doesn't support files over 4 GB. Or, there was no chance, I would format everything (especially USB) as UFS.

Lack of commercial quality disk tools like Disk Warrior if a true catastrophe happens is a problem too. Of course, fsck can do good things but after a true catastrophic filesystem issue, diskwarrior is a must. That was one of the things Professional Mac community had hard time explaining ZFS community.

As Apple was truly wise to completely document it down to a point you can even write a full feature defragmenter (iDefrag), HFS+ without journaling seems to be the best option. I am in video business and I have seen it deal with files way beyond 80GB without any issues. In fact, lots of OS X users who images their drives see it everyday too.

I don't know why journaling is not implemented, it is open and documented too. If a bit hassle happens, it sure deserves it since he deals with external drives which are just fit to journaling purposes.

Network?

[guruevi]

Really, you need a gigabit network and transfer files over it using AFP and/or NFS and/or SMB. First of all HIPAA requires you to encrypt your hard drives which most researchers won't do (it's too difficult). Then you also got the problem what happens if the researchers (or somebody else) leaves with the data.

Solaris and by extension Nexenta have really good solutions for this. You can DIY a 40TB RAIDZ2 system for well under $18,000. If you use desktop SATA drives (which I wouldn't recommend but ZFS keeps it safe) for your data you can press that cost to $10 or $12k.

I work in the same environment as you (neuroimaging, large datasets), feel free to contact me privately for more info.

Re:ext2 works. ntfs works.

[MachineShedFred]

If it's Mac OS X 10.6.x, you don't even n eed NTFS-3G, as the native NTFS driver has read / write capability. You just need to change the /etc/fstab entry for the volume to rw, and remount.

Re:NTFS

[FreonTrip]

Roses are red,

Violets are blue,

All of my mod points

Would belong to you.

UDF

[marquise2000]

I'm using a USB Disk formatted under linux with UDF (yep, it's not limited to DVDs, there is a profile for hard disks). It can be used without problems under OSX (even Snow Leopard)
 

the question of our age

[commodoresloat]

who will wooosh the woooshers?

NAS device

[linebackn]

A simple NAS enclosure or NAS device might be what you are looking for. You can get a single drive NAS enclosure, and add a drive, that you can carry around just like a regular portable drive. You can move it between networks and use any connection method the NAS device happens to implement (SMB, FTP, NFS, etc). Some even let you optionally connect it directly via USB or eSATA to access the file system directly, and some may have encryption or other security features as well.

Of course, check to make sure you have permission and that connecting things to your network does not violate any policies. If connecting a network device directly to the your network is not permitted then perhaps you can add a second, dedicated, network card to the computers.

Re:UDF IS ACTUALLY A SOLUTION

[marquise2000]

Ok everybody's occupied with surreal suggestions, but anyway:
*UDF* is quite awesome as a on disk format for LinuxOSX data exchange, because it has a file size limit around 128TB, supports all the posix permissions, hard and soft links and whatnots. There is a nice whitepaper summing it all up:
http://www.13thmonkey.org/documentation/UDF/UDF_whitepaper.pdf

If you want to use UDF on a hard disk, prepare it under linux:
1) Install uddftools
2) wipe the first few blocks of the hard disk, i.e. dd if=/dev/zero of=/dev/sdb bs=1k count=100
3) create the file system : mkudffs --media-type=hd --utf8 /dev/sdb (that's right, UDF takes the whole disk, now partitions)

If you plug this into OSX, the drive will show up as "LinuxUDF". I am using this setup for years to move data between linux and OSX machines.

Re:ext2 works. ntfs works.

[Anonymous Coward]

This is dangerous advice. There are numerous reports of instability and NTFS volume corruption when forcing 10.6 to mount NTFS volumes R/W. Apple seems to have turned NTFS write off by default for a good reason, it's not done yet.

Re:NTFS

[RobertM1968]

Windows doesn't play in here, it's OSX and Linux. Tossing NTFS into that would just be... wrong somehow.

Flamebait mod or not, there is a valid point. Though various NTFS drivers do allow read/write, the success isn't graven in stone. There are better alternatives in the Linux/OSX world. Keep in mind that losing this data becomes either costly (as in time=money, let's go make another set of copies to run to whatever office) or very bad (as in someone moved the files to the external instead of copying them) or both.

So, as good as the NTFS R/W drivers are getting, it's safer to use a file system that is known to be more stable and less error prone, such as HFS+ or UFS or one of the other suggestions. "Really good" shouldnt be an option in the medical world when "even better than 'really good'" is available, compatible, and easy to install on all systems involved.

Re:UDF IS ACTUALLY A SOLUTION

[mlts]

That is an excellent solution, and arguably the best to the OP's problem printed. UDF works on Windows, OS X, Linux. Even AIX is happy with it and can write to it. So an external drive with this on it should definitely solve the problem.

Been There, Fixed That

[DynaSoar]

We had almost exactly the same problem. Our fMRI work was done at University of Virginia on a Linux machine. naturally you don't want to tie up a $1500/hour data collection machine doing analysis. Our data was transferred immediately to the Neurological Institute to a multiboot machine. No patient data included at this point, so no HIPPA problems. The receiving box ran Linux initially since the analysis programs from NIH (primarily AFNI) were Linux based. Patient data got added here so HIPPA became an issue. The machine had multiple hard drive bays, all of which were removable, plug-and-play drives made from a kit that provided slide-in rails and a locking mechanism, otherwise were common, commercial drives. Externals would have been easier, but the guy who devised this had a rilly rilly good reason. I remember it was good, but not what it was. Anyway, the machine could boot other OSs, prep the drives, go back to the native Linux HFS+ and transfer/translate to the , it was transferred, the drive removed, packaged, and FedEx'd to the other analysis sites at Virginia Tech, NIH, and U.Va Wise. We were strictly experimental, no direct medical treatment, and so time was not an issue. With OS X being *nix, there's not a lot of reasons to go with one over the other except for convenience when it comes to what your data collection and analysis are running under. Unless yours run fine under OS X, I'd say stick with HFS+, and of course moderate that according to whether you have to share out the data and what those people are running. I wouldn't bother with supporting Windows, as they continually find new problems to have with large files. One comparison test showed no difference in analysis results, but they did have problems with Windows choking on the data files. Their test files were only 1.5 GB. ref: J Med Dent Sci. 2004 Sep;51(3):147-54. Comparison of fMRI data analysis by SPM99 on different operating systems. PMID: 15597820. My experience agreed with their results. As I said we had little call for Macs, so we didn't run enough of that to give a good test of whether it had the same kind of problems. Bottom line, we used what we needed to according to where it was going and what they needed it to be, but for our own use it made no sense to transfer it out of the OS that collection and analysis used, HFS. The system met with the approval of the biophysicist we worked with at U.Va, and he had been a grad student under Peter Fox when the latter developed SPM. OH YEAH: the good reason. If anyone else wanted to work with us, they didn't have to dig too deeply into techie stuff either hardware or software. We could send them a removable-drive kit to install, and send them a drive with bootable Linux, AFNI and data, all plug and play. If that might be useful to you (using externals instead of removables doesn't matter here) that's probably be another vote for HFS.