Fully Open Source NTFS Support Under Linux

lord_rob the only on writes "The Linux NTFS project has released a beta version of its fully open source userspace (using FUSE) 3G-Linux NTFS support driver. According to the developer, this driver beats hands down other NTFS support solutions performance-wise (including commercial Paragon NTFS driver and also Captive NTFS, which is using windows ntfs.sys driver under WINE)." That's right, writing to NTFS even works. Soon it'll mean one less recovery disk to keep around, I hope.

Comments

Great news.

[LinuxGeek]

This gives us another tool that can be used to repair windows systems that have been hit by some of the newest rootkits that can hide from detection when windows is running. Can't hide from a Linux boot disk and with complete write support, now these can be cleaned and studied more effectively.

Re:Great news.

[Tim C]

You also can't hide from a different installation of Windows that has the infected disk mounted. Rootkits hide themselves by hooking into the running kernel/fs drivers - inspect the disk with a clean install and they can't hide then either.

Of course, the more tools you have available to you, the better, and while it's very unlikely that a rootkit from one install can infect another as long as you're careful, it's *extremely* unlikely that it'll be able to infect a Linux install. That may change with time, of course - as with so many things, it's an arms race, and this one is unlikely to do anything but get hotter.

Re:Great news.

[Vo0k]

AFAIK most of their methods of protection would fail. Still, they could quite nicely hide in "alternate data streams" - every file or directory in NTFS chan have arbitrary metadata attached to it. Usually it's things like ownerships, permissions etc, but 'arbitrary' in this case means that besides the official metadata you can attach whole files making them invisible in the filesystem tree, existing in separate namespace, each file entry being a root directory for a whole invisible filesystem. So plain 'ls' won't show them. You need a tool that will examine each file, extract its metadata, discard the "standard" metadata and list whatever has been attached to files additionally.

Re:Great news.

[cnettel]

You can come with quite clever ways to hide data. The point is that the rootkit, to be an infection, must also have some way for a standard Windows routine to actually read that data and load it as code. In practice, it means that the real, non-rootkit-mangled, version of the registry will probably contain a reference, or that the normal data stream of some system binary will be changed as well.

Re:Great news.

[Thing 1]

You also can't hide from a different installation of Windows that has the infected disk mounted. Rootkits hide themselves by hooking into the running kernel/fs drivers - inspect the disk with a clean install and they can't hide then either.

Interesting approach: install VMware Server (free), install Windows into a VM (free if you have 2003--IIRC*, Microsoft allows 4 instances, 1 host and 3 virtual), then connect the physical drive to the VM. Not sure whether VMware will bypass the drivers and allow you complete physical access as I haven't tried it but that's one of the options when creating a new virtual hard drive.

You probably don't want to run the VM from the same drive that you attach to it, though... I haven't tested this, but it might be a nice option for investigating without taking down any services that may happen to be running on the potentially-infected PC.

* -- is this the sound made by a crashing car?

Nothing for you to see here. Please move along.

[roger6106]

Nothing for you to see here. Please move along.

Is Slashdot testing out the NTFS writing ability on their site?

Performance

[Reality Master 101]

Unless I missed it, I notice the performance numbers are only single process. I'm suspicious of this because user-mode filesystems (as under microkernel operation systems) typically crash and burn performance-wise under simultaneous load, not under single-user use.

I know that user-mode is easier to debug, but they really should turn this into a kernel module.

Re:Performance

[Ulrich Hobelmann]

Well, you can bet your ass that Windows's native NTFS is much faster than the Linux one, because they wrote the FS, and they have years of time to optimize the working driver.

Sure, user-mode will be a performance issue, but I think the context-switch + work is only necessary, when the kernel decides to either read data (on a cache miss) or write out its file buffers to disk. So if you don't use NTFS as your native disk (not sure how that'll work with permissions and stuff), you should really by fine. Maybe there's an option somewhere to turn on big read-write chunks (so that the kernel will always read/write huge blocks from the user daemon, which would make context-switch time negligible).

Re:Performance

[Schraegstrichpunkt]

Performance problems are a well-known fundamental problem with microkernel architectures that use user-mode processes.

It's a widely-believed myth, mainly due to the poor performance of bloated first-generation microkernels like Mach, although I suppose it probably also applies to Linux when Linux acts as a microkernel.

Google is your friend.

Just because Linus Torvalds thought something was impossible during the 1990s doesn't make it so, so I suggest you skip the infamous Linus vs. AST discussion from that time period.

The reality is that:

1. Microkernel architectures are hard to design. This is suspected as being the real reason why they are not very popular today.
2. Monolithic kernel architectures are prone to insecurities. There is just way too much privileged code, and too many failure scenarios.

Unlike Linus, some people are actually devoting much of their time to solving these problems. AST is one such person. See this page on the subject.

Re:Performance

[azhrei_fje]

Kernel drivers can not use virtual memory, explicit multithreading or advanced algorithms from MP, STL and hundreds of other C++ libraries.

Wow, I don't know where to start with this. First, mod parent down using -1, vague.

Have you written a filesystem? I have. The "virtual memory" you're apparently referring to is the process' virtual memory. This sounds like what you're trying to say is that the protection provided by process virtual memory isn't available in the kernel. And that's true. But that doesn't change the fact that the kernel can map any physical memory to any effective address. So the statement that the "kernel can not use virtual memory," is extremely bogus. ("Bogus" is a technical term. It means "completely wrong.")

Concerning "explicit multithreading," you must be referring to the idea that the kernel can't call pthread_create()? But there's no reason for the kernel to do so. Multi-threading in a user-level process is often done to achieve concurrency related to the delays that a single-threaded application would have if it had to wait for I/O to complete. The kernel doesn't have to do that at all. The kernel would queue up the I/O request, then continue on its merry way. When the interrupt from the device signals that the I/O has completed, a separate handler takes care of it. In a user process, threads are useful in order to modularize the I/O functions. In the kernel, they often aren't needed, since callbacks are used instead. Same functionality, different design technique. And even if they are needed for some obscure reason, all modern kernels (Linux included) support kernel threads. (My SUSE Linux 10 box currently has 19 kernel threads executing.)

The "advanced algorithms" you're referring to are probably coming out of user-space libraries. And in this regard, you're correct -- user-space libraries cannot (currently) be linked into the kernel and there is plenty of debate about whether such abilities should even be attempted. (The problem with user-space libraries inside kernel space revolves mostly around bugs and implementation deficiencies. The truth is that an algorithm that is mostly cpu-intensive probably could be loaded into kernel space using some kind of hack, and there are open source projects that are already working along these lines.)

In any case, there's no reason why those algorithms couldn't be executed inside the kernel. For example, take the find() generic algorithm from the STL (a macro from one of the libraries you mentioned). Why can't I use it? (The truth is I can.) And why can't I use the list class from the same library? I admit that linking large objects into the kernel could result in quite a bit of bloat, but there is not a technical reason that it couldn't be done. (Except in the case of C++ exceptions within the kernel. There is a group that has patches available for the Linux kernel that add support for C++ exception handling. With those patches, any STL code should be able to work in kernel space, although I've not tried it personally.)

It seems to me like the parent has read a magazine article and jumped to conclusions. Or perhaps they are even an experienced developer, but took huge liberties with the wording of their statement. But as "Captain Obvious," I felt it was my slashdot civic duty to clarify he issue. :)

Not Just Linux

[TheRaven64]

FUSE has been ported to FreeBSD, and it appears that the driver also works there.

No 64-bit

[Bill Dimm]

Looks like a great piece of work. One important note from the article:
Problem: Why doesn't the driver work on 64-bit and bigendian systems?
Answer: We have no resource for that. Neither hardware, nor workforce.
Status: Low priority.

Re:Awesome!

[lord_rob the only on]

A reply from the developper :

Currently I'm not interested in the kernel driver. It's a lost case for over a decade. Full read-write could be done in user space pretty fast and I can't see drawbacks, only benefits:

- NTFS is huge and complex, not for the kernel. Crash in kernel (hw error, corrupt ntfs, etc) and game is over. Crash in user space then just restart the service.
- kernel has a lot of limitations, restrictions which are all gone.
- fedora/redhat users have never ending hassles with installing the driver. Instead they could install ntfs-3g once and forget the issue forever.

Re:Awesome!

[ratboy666]

How to do it...

Assuming partition tables are "fubar".

#1 BACK UP ALL YOUR DATA. This is normally a sign of a failing drive.

#2 Download and burn a bootable CD of you hard drive vendors diagnostic kit.

#3 Run it, and "recertify" your drive. May take a couple of hours (and, you may just want to dumpster the drive, if your time is valuable). If the drive does not certify, discard it.

#4 Boot your system with Knoppix, or another recovery Linux system. Issue the command: "dd if=/dev/zero of=/dev/hda" (replace hda with hdb, hdc, hdd, etc. depending on which hard drive it is).

#5 Run Linux partitioning tool "fdisk /dev/hda". You *will* get a complaint about an improper partition table, which is ok. Partition, and write the new partition table.

#5b Alternatively, boot a Linux installation CD, and load Linux. Ignore warnings about "improper partitioning", and choose to have the partition table replaced.

The IMPORTANT steps are 1 to 3. If the partition table cannot be manipulated, it is an almost sure sign your drive is heading south.

Ratboy

ntfsmount != ntfs-3g

[Chris Pimlott]

ntfs-3g is brand new and it not the same thing as ntfsmount, which is what the current documentation covers. Please read the ntfs-3g announcement, which promises practically unlimited file creation and deletion.

It is good news ... But ...

[vtcodger]

Full NTFS compatibility in Linux is a good thing. There are a gazillion scenarios where it is necessary for users to get at Windows files from Linux or vice versa without moving stuff over a network.

But, keep in mind that NTFS remains proprietary and Microsoft can break it for newly written files any time it suits their business purposes to do so. All it takes is one update.

No one but me seems to care about this, but I think that the proprietary and undocumented nature of NTFS is an important reason why System Administrators need to have a workable exit strategy for Windows. They don't need to exit now. But in three or five or ten years if (when) Microsoft decides to lock in its user base, users should want to make sure that they have the option of being outside the door that Microsoft is slamming shut.

Re:It is good news ... But ...

[Martin Blank]

But, keep in mind that NTFS remains proprietary and Microsoft can break it for newly written files any time it suits their business purposes to do so. All it takes is one update.

Hypothetically, yes. However, there are few things that they -- or any OS developer -- are more paranoid about altering than the filesystem. You can recover from a bad driver, or a bad patch for most functions; recovering fully from a bad filesystem alteration may be nigh-on impossible, and Microsoft is going to think really, really hard before they go and do something that may result in lost data.

Re:It is good news ... But ...

[Antique Geekmeister]

No, Microsoft deliberately breaks compatibility. I suggest you look at the old FreeDOS lawsuits, and the Netscape lawsuits where Microsoft tried to pretend that NT Workstation couldn't work as a Netscape server, the oddness of Internet Explorer and their refusal to make it removable, and the weirdness they did to Kerberos that MIT sued them successfully over, and even take a look at what Active Directory does to DNS, and the current lawsuits about ODF in Massachusetts, and the recent EU lawsuit whaere Microsoft refused to document their software, instead publishing deliberately obfuscated and unusable documentation.

We always could write to NTFS

[r00t]

dd if=/dev/zero of=/dev/hda1

It's really fast, despite being in userspace, though it can still take a while because there is so much that it needs to do. Start it before you leave work, or before you go to bed.

As a side effect, your NTFS partition will finally be free of spyware. It's the only way.

Re:We always could write to NTFS

[RogL]

I say we dust off, ssh in and dd if=/dev/zero of=/dev/hda1 it from orbit. It's the only way to be sure.

Re:My thoughts on this...

[mh101]

Um, since when is 'interoperability' the same as 'lack of innovation' and 'stealing'? Nobody's trying to 'steal' NTFS to use in Linux. Rather, people are looking for a way to access their data from Windows that's stored on an NTFS partition. I don't think any Linux users would willingly give up EXT3 or ReiserFS for NTFS.

That's a pity

[Darkforge]

... because there is a really good reason to support NTFS in the kernel: so you can boot off of an NTFS drive. That would eliminate the need for Windows users to re-partition their drives when installing Linux, and allow for an easier dual-boot.

Now it's time to...

[ivan256]

In the grand tradition of open source NTFS drivers, this project has now reached the point in it's lifecycle where the developers abandon it and all future implementations start from scratch.