Slashdot Mirror
Open Source ExFAT File System Reaches 1.0 Status
Titus Andronicus writes "fuse-exfat, a GPLv3 implementation of the exFAT file system for Linux, FreeBSD, and OS X, has reached 1.0 status, according to an announcement from Andrew Nayenko, the primary developer. exFAT is a file system designed for sneaker-netting terabyte-scale files and groups of files on flash drives and memory cards between and among Windows, OS X, and consumer electronics devices. It was introduced by Microsoft in late 2006. Will fuse-exfat cut into Microsoft's juicy exFAT licensing revenue? Will Microsoft litigate fuse-exfat's developers and users into patent oblivion? Will there be a DKMS dynamic kernel module version of the software, similar to the ZFS on Linux project? All that remains to be seen. ReadWrite, The H, and Phoronix cover the story."
A file system is normally designed for one's own usages. A file system is entirely contained within your computer system (or in the event of a distributed file system, within computers under your control). What use then is "sneaker-netting" files between Windows, OSX, and Linux? Isn't this a network concept?
And over there we have the labyrinth guards. One always lies, one always tells the truth, and one stabs people who ask t
As far as I know it's part of OS X since Snow Leopard. But I could totally use the Linux support.
"The only way to win is not to play". If you use MS filesystem they're winning. Big operators like Google/Samsung/Sony (okay, maybe not sony) should be able to agree around some free and gratis FS for devices, and then use the courts if necessary to make sure it's easily available on Windows (I hear they have a store now).
What's wrong with ext2/3/4?
Microsoft will never allow one of its modern file system to have an "official" implementation on free operating systems.
What do you mean by 'official'? You mean from Microsoft? If so yes they probably won't create a Linux implementation, but that's cool because there's this project, there's Paragon and there's Tuxera if you really want exFAT on Linux, just like NTFS. Alternatively you could use the ext file systems.
There will be the threat of litigation or looming incompatibility and data loss through lack of public documentation.
We've had the NTFS driver for a long time.
Instead we should develop a simple and robust filesystem that's suitable for embedded systems and have it standardized.
Go for it, ext3/4 is probably a good start.
Obligatory XKCD
(Seriously though, patents on file system are bullshit.)
There's no -1 for "I don't get it."
One word: Windows
"Be particularly skeptical when presented with evidence confirming what you already believe." -
Standardise all you want. You should know what'll happen. Windows will not support it out the box, and if Windows doesn't support it, that filesystem is effectively dead. Who is going to want a USB stick formatted so it won't work on the operating system running on upwards of ninety percent of desktops and laptops?
First they'll give enough time for it to get established to the point of being considered an essential for any functional desktop.
*Then* they'll start suing.
As the name clearly states, this is a FUSE implementation of exFAT, i.e. userspace. In which case DKMS is as useful as a fork for soup.
So not only we get the news two days after Phoronix [1], but the poster has no idea on what he's talking about.
[1] http://www.phoronix.com/scan.php?page=news_item&px=MTI3OTQ
The courts do not have that power. Maybe - maybe - in a full antitrust case, but the last time MS was involved in one of those... well, United States v Microsoft was filed in 1998, and concluded in 2002. Four years, and it only finished because MS settled it on terms quite favorable to themselves. Legislative action could do it, but good luck out-lobbying microsoft, not to mention all the economic conservatives screaming about how the commies are trying to steal the hard work of a good American company.
Those are not supported by Microsoft Windows right out of the box so that are not readily suitable for use in flash drives and SD cards.
Microsoft has already won by having ExtFAT part of the SDXC spec, so every big SD card comes with it. The only thing the Open Source world can do is damage control by implementing it and thus staying useful.
NTFS on linux was created through many years of hard work reverse engineering the filesystem from no documentation - what little MS had published was only available under licenses that would render it useless for open-source development. That it works at all is impressive, that it works so well is a small miracle. Even now, NTFS support in linux has to be via the NTFS-3G userspace filesystem - full support was never included in the kernel itsself, only read-only access. That may well be the future of linux and exFAT: It works, but exists in a legal grey area where MS could unleash the lawyers on a whim and requires untidy hacks to get around legal problems.
Instead we should develop a simple and robust filesystem that's suitable for embedded systems and have it standardized. Right now there simply isn't an alternative to the FAT filesystems.
It would seem that Samsung has already done just that: http://www.h-online.com/open/features/Kernel-Log-Coming-in-3-8-Part-1-Filesystems-and-storage-1788524.html
Linux now supports F2fs (Flash-Friendly File System), a filesystem that was introduced by Samsung developers in October. It is designed for flash storage media that uses a more basic Flash Translation Layer (FTL) than SSDs for desktop PCs and servers – for example USB flash drives, memory cards and the storage media that is used in cameras, tablets and smartphones.
Am I allowed to use this implementation?
Depends on what you want to use for: as a form of expression, you should be able to. Use the binary form to read/write, all depends on the MS patents and whether or not MS grants you a license.
Will Microsoft litigate fuse-exfat's developers and users into patent oblivion?
Regarding developers: the software is posted as source code with instructions on how to install them from source. Being source code, is a form of expression, protected by copyright. As such, can a commercial entity try to block the dissemination of the "speech" that the source code constitutes?
Mind you, any existing patents should not play any role into it: after all a patent is a public disclosure of methods/constructs that constitute the invention (the text of the patent is not copyrighted), so the source code should not be anything but an alternative form of expression of the same.
Regarding users: yes, using the compiled binaries would violate the temporary monopoly granted by any existing patents. However, I can't imagine any corporations starting to track which hobbyist home users:
1. downloaded the source code - should not be, per se, illegal - the copyleft license allows you to do it and the patent should not trump the copyright.
2. for each of them, ask for a discovery to see if that source code has been compiled - again, compilation should not be illegal, I'm obtaining a derivative form of expression and the GPL copyright license allows me to do it
3. use the binary - this is the only step that would violate the patent
Questions raise, answers kill. Raise questions to stay alive.
Put the driver for the standardized/interop filesystem in a (comparitive small 1%) fat16 partition* on the same media. Include nice installers and use the autorun features of the OS. Problem solved.
*: I had an older 512Gb USB drive that included the drivers for retarded windows versions on a seperate device, it emulated a cd with iso9960 of UDF.
One word: Windows
ext is open, you can implement it on Windows (in fact it's already been done).
Ext2/3/4 sucks as an interchange format. In short, it does too much. Any filesystem sufficiently complex to support real workloads is going to impose an excessive implementation burden for sneakernet. The bizarre thing is that we have a minimalist filesystem that can represent the file model with fidelity (large files, unicode names, etc) that is implemented in every modern OS: UDF. If it can read DVDs, it can read UDF and every general purpose OS released in the last decade can write to the appropriate version, 2.01. Not for nothing is it called the Universal Disk Format.
The real mystery is how did Microsoft con an industry into paying for such a lousy alternative to UDF. SDXC requires exFAT, so every new camera and anything else that hopes to read these high capacity sdcards has to cope with licensing requirements. WTF.
The kernel has ( or had, I haven't built one in a while ) experimental write support since at least the 2.6.18 branch or prior... I don't remember exactly when it was introduced. It is completely useless[1], but it is there.
That said NTFS-3G really is the way to go. It also, as an added bonus, fits the UNIX philosophy: "Do one thing and do it right".
[1] The in-kernel write support for NTFS only allows you to write to an existing file, and only allows you to write the same amount of data as the exact files size and name on the disk. Not very useful since you can not even create files.
To err is human; effective mayhem requires the root password!
fuse-exfat is also an userspace driver like ntfs-3g. If US-based distros like Fedora ar able to ship with ntfs-3g installed by default, they might be able to do the same with fuse-exfat, unless Microsoft closed the legal loophole used for ntfs-3g.
Those are not supported by Microsoft Windows right out of the box so that are not readily suitable for use in flash drives and SD cards.
A lot of devices are not supported by Microsoft Windows right out of the box, that's hardly causing problems, that's just an excuse.
You can call FAT and its variants a lot of things, but "modern" isn't one of them.
The world's burning. Moped Jesus spotted on I50. Details at 11.
NTFS support in linux has to be via the NTFS-3G userspace filesystem - full support was never included in the kernel itsself, only read-only access.
It doesn't have to be, Linux maintainers don't want patent-encumbered specs implemented in the kernel, that's their choice, distro vendors may feel differently.
Ext2/3/4 sucks as an interchange format. In short, it does too much. Any filesystem sufficiently complex to support real workloads is going to impose an excessive implementation burden for sneakernet.
Format disk, use as normal. Hardly an excessive burden. But yes UDF works just as easily.
Sadly, you'd have to adjust the thermostat in hell first.
"Be particularly skeptical when presented with evidence confirming what you already believe." -
Pretty sure the developers aren't doing it for "damage control". They're doing it because they enjoy it.
There's UDF. It works on Windows out of the box (most impotant feature), supports Unicode, large files and volumes, UNIX / DOS / OS2 / Mac file attributes and special files (symlinks, devices etc), and extended attributes. It works much faster than FAT on flash drives. And its specification is freely available. But it might be covered by patents too.
The fact that this isn't in the kernel and that device manufacturers can't ship it remains a serious problem for Linux.
Though there are several solutions, none of them are ideal. When you're running Windows 7 x64, it gets even more hairy. Here's one discussion thread about the problem and options. So, I'd be unwilling to call it "already been done." The problem is that the Windows file system "API" is less than functional and a pain in the ass under the best of circumstances. From what I have read, the universal "best solution" is to boot from a Linux (any) LiveCD if you want to work with ext# partitions.
"Be particularly skeptical when presented with evidence confirming what you already believe." -
Explore2fs works fine, doesn't support ext4 but if that's your thing then the software is under GPL so open source to the rescue.
You mean the 4GB file size limit of FAT32 never causes problems? Microsoft, Apple, the Linux community, and possibly Google really should come together and create a new, open, and license-free file system spec that they all agree to support in their respective operating systems. They could come together every ten years or so to create a new spec. Of course, Microsoft wouldn't want to do that. They make too much money through threatening companies that make their own implementation of one of Microsoft's proprietary file systems.
You mean the 4GB file size limit of FAT32 never causes problems?
No, i mean A lot of devices are not supported by Microsoft Windows right out of the box, that's hardly causing problems. You really think people only use things that are supported by Windows out of the box?
Explore2fs supports read-only and does not have Windows Vista/7/8 support. My statement still stands.
I can write code but, I'm not a programmer so any attempt I make to work on it will just make things worse.
"Be particularly skeptical when presented with evidence confirming what you already believe." -
Explore2fs supports read-only and does not have Windows Vista/7/8 support.
Since you won't bother to read your own link I'll quote it for you here:
Works great for me in Windows 7 x64. Provides decent transfer speed and apparently supports both read and write ability. Ignores permissions so you can get access to every file on the partition.
In any case it's not that it can't be done, it's that nobody wants it, people just write utilities that are just enough for what they need, if you cobble the functionality together you have a workable solution. It's all open so either learn to code, pay somebody to build it for you or forget it altogether and pay for one of the existing exFAT proprietary solutions, all the options are there.
It's annoying using a file system with file ownership on a flash drive, because chances are the computer I plug the flash drive into has an entirely different set of user IDs that don't match up to the flash drive's files' ownerships. I wish there was an easy way to mount an ext filesystem with all of the files owned by a specific user id (such as the id of the active desktop user when I plug in the flash drive). I wouldn't be surprised if there already is a way, but it should be do-able via the UI and not require root access.
What use then is "sneaker-netting" files between Windows, OSX, and Linux? Isn't this a network concept?
Not if the network is cost prohibitive. Good luck transferring 10-gigabyte files over cellular.
ext is open, you can implement it on Windows (in fact it's already been done).
So if I have a USB flash drive formatted in ext, how do I load the ext driver onto an Internet-disconnected PC running Windows so that I can use the flash drive with that computer?
A lot of devices are not supported by Microsoft Windows right out of the box, that's hardly causing problems, that's just an excuse.
This, unlike the case you're thinking of, does cause problems. People normally use a USB flash drive to copy the driver for one of these "devices [that] are not supported by Microsoft Windows right out of the box" from the computer with the fast Internet connection to the computer in another building with the printer, scanner, or other similar peripheral. But when the flash drive itself is the peripheral, it creates a chicken-and-egg situation. Or do you believe that anybody who doesn't carry two flash drives, one large one to store the data and one small one to store the driver for the large one, is just making excuses?
You want to support SD cards larger than 32 GB?
No, not at least until you can prove it's absolutely necessary. Why can't a future device design just drop support for SD cards and use UDF or Ext formatted USB flash drives instead? As a widely respected first-century teacher might have put it: "If [SD support] causes you to stumble, cut it off."--Mark 9:43.
If that's your problem then a tiny partition with a utility or driver would work fine.
Jelly Bean's Play Store introduces DRM and that's incompatible with GPLv3.
I wasn't aware that the APK encryption in JellyBean was mandatory.
I had a sig once. It was lost in the great storm of '09.
It's not really a replacement for FAT, though. F2FS is very obviously designed for flash memory, whereas FAT is media-independent.
I had a sig once. It was lost in the great storm of '09.
It works on Windows out of the box
Write support only works out of the box on Vista and later. Not a huge problem to work around, but it's there
UDF certainly looks like the most appropriate candidate for a truly universal file system, though
I had a sig once. It was lost in the great storm of '09.
Crappy xttrs support (4kbs if enabled)
https://en.wikipedia.org/wiki/Inverted_totalitarianism
So flip the removable bit.
I thought removability was part of the USB device controller, not something stored in the file system, and there was no standard way to "flip the removable bit" that works across brands of flash drive. This answer confirms my suspicion. Or were you recommending that people investigate which USB flash drive brands support end-user control of the removable bit before buying the drive in the first place?
Or use FAT32 instead.
Windows won't format FAT32 bigger than 32 GB, and surpassing the 32 GB limit is SDXC's reason for existence. As I understand it, the only technical difference between SDHC and SDXC is the file system that a device is required to support.
I can think of two uses off the bat for kernel-level NTFS overwrite-existing-file support:
* Use an existing fixed-size pagefile.sys as a Linux swap file on a dual-boot machine. Just be sure the system isn't hibernated before trying this trick. I don't think I want to take the performance hit of doing this in user-space.
* File-blanker. OK, this can be done in userspace, but I can contrive of oddball situations where it might be better to do it in kernel space.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
* Reasonable file- and path-size limits that won't break common usage scenarios.
* Support for common meta-data that everyone expects to be there, like date-stamps.
* The speed that you would expect from a simple file system (#2 above).
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Patents on software are bullshit. That's the real reason why Nvidia doesn't have an open driver that patent trolls can look at.
> Who is going to want a USB stick formatted so it won't work on the operating system running on upwards of ninety percent of desktops and laptops?
I, for one, would. Right now, I use HFS+ for everything (internal, external) but if OS X could read/write something that would also work on Linux, I'd be happy to use that. If it wasn't readable on a Windows machine, it wouldn't make a lick of difference to me.
How about when you are not admin? Installing software, especially drivers, are not always appropriate or even possible.
This reverse-engineered NTFS - even if it has no space in Linux, can it be used in ReactOS, which is not using NTFS due to avoiding violating MS patents there? Actually better - if they can produce a reverse engineered 64-bit equivalent of NTFS, name it something else and make it the native file system of ReactOS?
The Zevo ZFS and ZFS on Linux projects might be of use to someone with your set of requirements. They provide read/write access to ZFS filesystems for OS X and Linux, respectively. (AFAIK, a native Windows implementation of ZFS does not exist.)
That sort of thing worked in the win9x era when everyone had autorun enabled and there was little lockdown of systems. Nowadays it's going to mean your device can't be used with all the locked down desktops found in corporations and educational institutions.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
FAT is in the kernel (presumably reverse engineered). That is supposedly covered by patents. It should not really be any more dangerous to put ntfs, exfat or zfs in the kernel.
(ZFS is slightly odd, in that there is already code that could easily be dropped in if it were not for licensing. It would need some to reimplement it under a GPL licence, but i guess while there is the faint possibility of the original code being relicensed no one will want to put in the work)
+1
if you plug a drive into another computer you should have no expectation that the computer will respect the permissions, so what is the point of having them. it should be possible to make a udev rule that 777s any external disk when you mount it. or hack ext4 to ignore permissions on external drives.
Yes, but only tech savy people will install the necessary drivers. Most likely your ext4 formated USB drive will only work with the SOME machines you administrate(things that you can't customize the OS/firmware such as game consoles, external HDD compatible TVs, several professional measurement instruments are some of the devices I use that have only some for of FAT or NTFS support). Good luck in the real world. I HATE microsoft because of that. Using the "monopoly" on the OS to basically dictate which file systems hardware makers all over the world will use is, IMO, as bad as having trying to force everybody in to developing only for IE6 back in the 90s. No, scratch that, it's actually worse because you have to pay patents. And virtually nobody does anything about it.
The in-kernel write support for NTFS only allows you to write to an existing file, and only allows you to write the same amount of data as the exact files size and name on the disk.
This is not quite accurate: support for extending existing files w/ the in-kernel driver has been available for some time (ca. 2005). Apparently it can fail to extend the file if it is excessively fragmented, however.
so when I need to use large files on multiple computers it's faster to carry an external hard drive arround than to try and use the network.
And that external hard drive can be formatted UDF, as long as you don't need to write to the drive from a Windows XP machine.
I guess I could carry a laptop arround and setup a network connection directly with the machine I wanted to use the file on but that is a lot bulikier and more hassle than just plugging in an external hard drive.
That or an external hard drive in a NAS enclosure, so that any computer that speaks FTP or SMB can read and write its files. Or an external hard drive in a USB enclosure that speaks MTP, so that any computer that speaks MTP (such as any Windows PC or any Mac with the MTP class driver) can read and write its files. The difference is that both NAS and MTP work on the level of files, unlike Mass Storage that works on the level of disk blocks, and the host OS need not be aware that the enclosure is using Ext behind the scenes.
Sorry, I went one link deeper and didn't specify. I linked to the site I did as it had more options if people wanted to investigate.
According to the Explore2fs official website, the current version only supports read-only and only up to Windows XP. There is a beta of version 2 (v0.7) that has "Supported by all versions of Windows (Vista is still Work In Progress)" listed. I'm not saying you can't make it work or that it won't work in many scenarios. But, it's not ready for prime time in a production environment.
"Be particularly skeptical when presented with evidence confirming what you already believe." -
If it's that old I'm guessing it was a 512MB drive, not 512GB.
But it is doable, everything is open and even many of the utilities (like Explore2fs) are open source so there's even easy starting points. Really the only thing stopping it is the fact that nobody uses it, which is reasonable given that 98%+ of people use Windows or OSX (which support exFAT), then of the remaining 2% there are at least 2 proprietary licensed exFAT solutions and now a free and open one, how many people need a solution to this problem and don't fit into any of those categories? Probably none, so it's not surprising that the ext solution hasn't gained traction, there's no technical barriers, it's not even particularly difficult, it's just pretty pointless given there is no audience for it.
Yes, but only tech savy people will install the necessary drivers.
And 98%+ of people will be using exFAT on Windows or OSX, of the remaining needing the ext solution. It could be implemented in the same way that drivers are done for other peripherals, it could even be automatically included on a small partition on the device or something like that, this isn't a problem of technical feasibility, it's that nobody would ever use it, they'll just use exFAT or alternatively FAT32 (and on FAT32 if you have >4GB files you just split them with an archiver).