Sony Beefs up FAT for Consumer Devices

An anonymous reader points to a report at LinuxDevices which says that "Sony has created an enhanced version of the vFAT filesystem that it says works better in Linux-based consumer electronic devices with removable USB mass storage devices. Unlike vFAT, the xvFAT filesystem will not induce a kernel panic if a USB storage device is removed during a write operation, Sony says," and writes "For now, xvFAT is a patch to the Linux 2.4.20 source tree maintained by CELF, an industry group of consumer electronics giants working to improve Linux for CE devices. Sony intends to submit the filesystem for inclusion in the mainstream 2.6 Linux tree as well."

Comments

Still won't buy Sony products.

[matria]

This is the same company that sues DVD decryption software authors, sues restaurant owners who's name happens to be Sony, sues people using old Sony cases for iPod mods.

So apparently it's OK to "vigorously defend their IP", while blatantly violating everybody else's. I wonder if the same big bucks and lawyers that force little people to bow to their corporate steamroller had anything to do with voiding Microsoft's patent of the FAT filesystem?

Don't misunderstand me; pulling out that particular stupid patent after all these years was a dirty stunt, and certainly deserved to get shot down, but that doesn't make Sony any cleaner. Two rabid pit bulls aren't any better than one.

Re:Still won't buy Sony products.

[poopdeville]

Oh come now... xvFAT is amazing it will- - optimize seamless communities - generate vertical e-services - everage synergistic convergence and best of all - engage e-business content Perfect solution

Who to believe

[superpulpsicle]

You can believe Sony marketing that vFAT is truely more enhanced for electronic devices.

You can believe Sony doesn't want to pay M$ for using FAT. Therefore finding a need to innovate alittle.

You can believe Sony will probably not go very far with yet "another" standard it created.

The choice is yours...

How many...

[Creepy Crawler]

Write operations does it do to the file allocation table when a decently sized file is copied?

Whats the average throughput compared to a baseline of linearly read data?

Could it be used on CD's and the like instead of iso9660fs and have the main OSes capable of reading them?

Whats the CPU% used for reading 1 MB sequential data per second?

Whats the license (if dual licensed or such) and are there any SOny patents that they might try to stifle this later?

Why cant you prevent Panics from removing vFat utilizing devices? Shouldnt have Linux came up with a way to gracefully determine 'dirtiness' and then dump the kmod gracefully?

I know some questions sound paranoid, but this is Sony we're dealing with. UMD, mem-stick, and god knows how many other things they've encumbered with crap and DRM have proved them one way. This proves them slightly the other....

Very weird company. Hurt with one hand, heal with the other.

Re:How many...

[TopSpin]

Creepy Crawler:
Why cant you prevent Panics from removing vFat utilizing devices? Shouldnt have Linux came up with a way to gracefully determine 'dirtiness' and then dump the kmod gracefully?

Foolhardy:
What does the filesystem have to do with crashing, other than the quaility of the driver? i.e. what do the on-disk file structures have to do with having a kernel panic?

Good questions. You've just stumbled into a significant flaw in *nix generally.

Linux, begotten of Unix, does not subscribe to the notion of transient filesystems. Behavior is undefined when filesystems vanish suddenly. It seems obvious enough; the kernel should block IO activity, flush buffers, unmount and return errors to users that are attempting IO to the now missing filesystem. Whatever "damage" occurs to the data (as opposed to filesystem metadata) is, rightly, the users problem. Unfortunately, this is not what happens.

What does happen falls under the euphemism "implementation defined." A good example is evident with NFS; *nix admins have been independently discovering this for years. If an NFS mount vanishes, *nix processes often hang indefinitely with no means of recovery. Various "soft mount" hacks appeared to accommodate the real world where network problems exist. Again, the actual behavior is not consistent; "soft mounts" are not always honored and obscure things like NFS versions or various "modes" of IO factor into why or why not.

I believe that in the early days the need to optimize IO led to designs that made no allowance for transient filesystems. This design propagated itself into POSIX, where behavior was left undefined. Even today you find crazy things like kernel panics when a FAT filesystem does something other than remain perpetually mounted. There is no "correct" thing to do and developers, hesitant to start inventing policy where none exists, go on being oblivious to the problem.

The fact is that a large percentage of "important" filesystems are transient. Remote storage, removable storage, etc. host valuable data, while permanently attached storage provides only basic machinery.

Sony, stuck trying to make transient vFAT filesystem hosting devices play nice with Linux, has stepped in and attempted to address the problem. *nix will be dragged kicking and screaming into the modern era of transient filesystems. Unfortunately, Sony's pragmatic, special case solution does nothing to address the larger problem, and whatever solutions emerge for all the other possible cases probably will be/are inconsistent in both implementation and behavior.

Blame the *nix folks who, 30 years ago, failed to anticipate hot pluggable keychains with hundreds of megabytes of storage.

Re:How many...

[david.given]

You've just stumbled into a significant flaw in *nix generally.

Everything you've said here is correct, and I agree with you, but you haven't mentioned the fact that in general, dealing with transient filesystems is an enormously hard problem on any real OS. There is no quick fix for this.

The problem is that you have to make sure that the filesystem on disk is consistent when the media is removed --- but by the time you know that the media is being removed, it's too late to do anything!

Unix deals with this problem by simply refusing to deal with it: it requires you to dismount all filesystems before disconnecting the media. Which is fine if you're dealing with hard disks, but less fine on USB devices and floppies. (There's a good reason why most serious Unix hardware have software floppy drive eject mechanisms.)

(Unix has the particularly unpleasant issue of the unified VM and I/O system; what do you do if you want to page in a block from a file system that's gone away? Seg fault? Block until it comes back again? Wave your arms in the air and run around in small circles? Different implementations do all three...)

Windows and DOS attempt to deal with the problem by using write-through cacheing on anything it thinks is transient. This kills performance. (Try switching off write-through cacheing on your floppy disk sometime.) But even Windows wants you to dismount USB devices before removing them.

Even CP/M had a variation of this problem --- there were specific system calls to detect disk changes and discard its caches. You were supposed to call this every time your program stopped for user input. Not all programs did, which meant that if you changed disks at the wrong time, you could end up with a corrupted disk...

Re:How many...

[Foolhardy]

On Windows 2000 or later, there is a special PNP IRP that is sent to devices that have been yanked: IRP_MN_SURPRISE_REMOVAL. All removable devices must support this IRP and it must succeed. Devices are to release any resources used by the device, fail all pending and future operations on the device (other than close/cleanup) and await a IRP_MN_REMOVE_DEVICE once all handles have been closed for final cleanup. This IRP is generally handled by the volume device, below the filesystem. The devices, including the filesystem, play dead until all handles are closed.

I pulled a USB flash drive while Explorer was copying a file to it. Explorer immediately gave me an ugly dialog box about not being able to find the file path with no option to retry or continue. I'm not sure what happens with memory-mapped file IO, but I know that when a pagefile becomes inaccessible, any thread that hits a hard fault hangs.
Paging files aren't supposed to be located on removable devices, because pagefiles can't be closed at runtime. Besides, the kernel would die with a KERNEL_DATA_INPAGE_ERROR if it couldn't retrieve a kernel page from a removed disk.

2000 enables write caching on removable devices by default, requiring you to use the "Safe hardware removal" applet. Microsoft found that too many people were simply yanking the things, so they made write caching disabled by default on XP. You can change the status of write caching on any volume in hardware->properties->policies.
It is safe to pull an idle device without using safe removal when write caching is disabled.

Crash???

[Foolhardy]

Unlike vFAT, the xvFAT filesystem will not induce a kernel panic if a USB storage device is removed during a write operation, ...

The kernel normally crashes when you yank a block device on a USB bus that uses vFAT?
The FS can't just fail the pending write operations? It has to kill the kernel?

What does the filesystem have to do with crashing, other than the quaility of the driver? i.e. what do the on-disk file structures have to do with having a kernel panic?

I mean, that's what xvFAT is, a different set of disk structures, isn't it? (not just a different driver)
There's really no way to make the current vFAT driver recover safely with the current FAT disk structures?

Re:Crash???

[KiloByte]

It's a mount option -- in the case of data corruption, it's usually safer to go down instantly rather than continue. However, I don't expect anyone but the worst lunatics to set it on a FAT filesystem, especially one on an USB device.

Having the kernel crash (as opposed to a panic) due to a bug in a filesystem driver is another story. I've once discovered that a maliciously malformed filesystem can send everything into the la-la land. This is where Hurd's separation of kernel structures would be useful.

Just trolling, ignore me.

[LittleBigLui]

But will you be able to cp stuff off that filesystem or is it write-only?

Re:why...

[10101001 10101001]

Like the summary explains, vfat (ie, fat) is used in all sorts of consumer electronic devices because it is (well, was) the most common filesystem for desktop users. Add to that, near ever desktop OS supports reading/writing fat and it's not much of a surprise that it'd be the common filesystem for ubiquity in data access on portable devices. The only other filesystem that'd make sense would be udf. But not every OS (read, Windows) readily supports that through usb, floppy, etc.

And if you wonder why they don't just use a data syncing program instead of exposing the filesystem itself to users, the simple answer is that it's much easier and desirable to the computer to just plug in a usb device anywhere and access it as a mass storage device than it is to have to install software everywhere for every device that you ever plan to connect to any computer you plan to use. Overall I like it this way since it means that if I'm willing to trade accessibility I can use whatever filesystem I please. But odds are I'll still use vfat because I really dislike the idea of not being able to access it wherever I go.

Re:Wow, Sony just gained major cool points for tha

[tepples]

It would have been even cooler if they would have just used an already implemented open source file system and then wrote a driver for the Windows guys

Except the installable file system layer in Windows 2000 and XP has a reputation for being poorly documented. The file system headers also cost money, though an order of magnitude less than it cost in the Win2k era. Sony could pay this, but this expense is why you don't see an IFS port of Ext3, Reiser4, or any of the other popular Free file systems.

Re:why...

[bersl2]

vfat is the greatest common denominator of file systems.

The number of filesystems Windows supports is pathetic, because it boils down to FAT32, NTFS, ISO9660, and SMB/CIFS. Your options are really quite limited.

The FAT patent was invalidated.

I have to admit, I'm skeptical

[croddy]

I'm skeptical that yanking a vfat-formatted usb-storage device out of the port while I'm in the middle of a write operation is going to cause a kernel panic.

I'm gonna give it a shot.

Right now I'm going to start copying a large file to my thumb drive, and once it's got 30-40mb done, I'm going to pull out.

Wait for it!

Good news, everyone! All I got was an error message from GNOME -- "I/O Error while copying file foo.avi. Would you like to continue? Skip/Cancel/Retry"

I'm gonna stick the drive back in and tell it to continue -- stay tuned!

Holy crap, it picked up all on its own.

Wait...

Yep, it just passed an fsck.

Sony, what are you smoking???

Re:Vaporware

[tomstdenis]

Something like ReiserFS doesn't scale well at the low end. On a 128M memory stick [iirc] the file system takes something like 16-20MB off the top. That's 15% of the drive gone instantly [compared to the 5M that FAT takes].

Granted if you have a 1GB memory stick the 20M or so that Reiser takes is less of a pain and the gains you get from the stability are more worth it...

Tom

Re:Wow, Sony just gained major cool points for tha

[Directrix1]

Here is a ext2fs driver for windows. Not currently the most top of the line file system for Linux, but still a pretty nice addition (you know if you like dual boot and don't want your partition to have any security respected on it :-P).