Fully Open Source NTFS Support Under Linux

lord_rob the only on writes "The Linux NTFS project has released a beta version of its fully open source userspace (using FUSE) 3G-Linux NTFS support driver. According to the developer, this driver beats hands down other NTFS support solutions performance-wise (including commercial Paragon NTFS driver and also Captive NTFS, which is using windows ntfs.sys driver under WINE)." That's right, writing to NTFS even works. Soon it'll mean one less recovery disk to keep around, I hope.

Comments

Great news.

[LinuxGeek]

This gives us another tool that can be used to repair windows systems that have been hit by some of the newest rootkits that can hide from detection when windows is running. Can't hide from a Linux boot disk and with complete write support, now these can be cleaned and studied more effectively.

Re:Great news.

[maxume]

Are the root kits so insidious that it is dangerous to put them in a throw away windows box as a non boot disk? That would seem like a fairly effective way to study them. I don't think I would want to go through that as a repair procedure, but for research, a disk swap seems easy enough.

Re:Great news.

[Tim C]

You also can't hide from a different installation of Windows that has the infected disk mounted. Rootkits hide themselves by hooking into the running kernel/fs drivers - inspect the disk with a clean install and they can't hide then either.

Of course, the more tools you have available to you, the better, and while it's very unlikely that a rootkit from one install can infect another as long as you're careful, it's *extremely* unlikely that it'll be able to infect a Linux install. That may change with time, of course - as with so many things, it's an arms race, and this one is unlikely to do anything but get hotter.

Re:Great news.

[Vo0k]

AFAIK most of their methods of protection would fail. Still, they could quite nicely hide in "alternate data streams" - every file or directory in NTFS chan have arbitrary metadata attached to it. Usually it's things like ownerships, permissions etc, but 'arbitrary' in this case means that besides the official metadata you can attach whole files making them invisible in the filesystem tree, existing in separate namespace, each file entry being a root directory for a whole invisible filesystem. So plain 'ls' won't show them. You need a tool that will examine each file, extract its metadata, discard the "standard" metadata and list whatever has been attached to files additionally.

Re:Great news.

[Anonymous Coward]

Can't hide from a Linux boot disk and with complete write support, now these can be cleaned and studied more effectively.

They can't hide from a Windows boot CD either, and given that NTFS is a proprietary file system (i.e. open source drivers will always be playing catch-up), I'd be more inclined to trust the official NTFS drivers on a Windows boot CD.

The Windows Vista (beta) setup disc boots to a live system, which can be used for repairs and whatnot, but with older systems like XP, it's a bit of a hassle, in that you have to build your own CD (there are tools available for download that make this easy for the technically inclined, but for novices it's still difficult). In any case, anyone with the competence to use a Linux disc to repair Windows should have no problem building a live Windows XP (or 2003) disc, and using it for repairs.

What this driver is good for is accessing data on a Windows partition from Linux. I primarily use Windows, so I like to keep my data on NTFS partitions, since all the Windows security attributes are meaningful. Although drivers for common Linux/BSD file systems are available for Windows, in my experience, the security attributes can only really be meaningful in one OS or the other, so I'd prefer to keep them meaningful in the one I use most, i.e. Windows. Being able to read and write from Linux, however, is a nice feature to have, especially with good performance, and either a reliable KM driver or a UM driver that's at least reliable enough not to corrupt the file system.

Re:Great news.

[cnettel]

You can come with quite clever ways to hide data. The point is that the rootkit, to be an infection, must also have some way for a standard Windows routine to actually read that data and load it as code. In practice, it means that the real, non-rootkit-mangled, version of the registry will probably contain a reference, or that the normal data stream of some system binary will be changed as well.

Re:Great news.

[Schraegstrichpunkt]

You also can't hide from a different installation of Windows that has the infected disk mounted.

In theory (assuming a sufficiently naive theory) that is true. In practice, all it takes is Explorer and something like a few WMF files. Heck, Explorer renders HTML for its thumbnail view, so it probably wouldn't be too hard for an attacker to find an exploitable bug somewhere in that code path.

Re:Great news.

[Thing 1]

You also can't hide from a different installation of Windows that has the infected disk mounted. Rootkits hide themselves by hooking into the running kernel/fs drivers - inspect the disk with a clean install and they can't hide then either.

Interesting approach: install VMware Server (free), install Windows into a VM (free if you have 2003--IIRC*, Microsoft allows 4 instances, 1 host and 3 virtual), then connect the physical drive to the VM. Not sure whether VMware will bypass the drivers and allow you complete physical access as I haven't tried it but that's one of the options when creating a new virtual hard drive.

You probably don't want to run the VM from the same drive that you attach to it, though... I haven't tested this, but it might be a nice option for investigating without taking down any services that may happen to be running on the potentially-infected PC.

* -- is this the sound made by a crashing car?

Re:Great news.

[cryptoluddite]

I had a disk with a problem of a couple bad blocks that cause NTFS to freak out. Connecting it to another Windows machine cause it to freak out also (as in ntfs driver hosed and left 2nd system's filesystem corrupt). So I wouldn't count on examining a Windows rootkit from another Windows system if I had linux available.

Re:Great news.

[TheRaven64]

You already have such a tool in Captive NTFS. True, that requires the Microsoft driver, but if you're going around fixing NT-based systems, you have that.

You would, of course, have to have a copy on your recovery disk. The first thing I would expect a rootkit to do is patch the NTFS driver, so relying on the infected one would not be a good idea.

Re:Great news.

[Jeremy Allison - Sam]

No, you can execute directly out of the alternate data stream so long as you end the stream name in .exe....

See here:

http://www.infosecwriters.com/texts.php?op=display &id=53

for details. It shows a process listing with myfile.txt listed as a running process.

Scary stuff :-).

Jeremy.

Nothing for you to see here. Please move along.

[roger6106]

Nothing for you to see here. Please move along.

Is Slashdot testing out the NTFS writing ability on their site?

Performance

[Reality Master 101]

Unless I missed it, I notice the performance numbers are only single process. I'm suspicious of this because user-mode filesystems (as under microkernel operation systems) typically crash and burn performance-wise under simultaneous load, not under single-user use.

I know that user-mode is easier to debug, but they really should turn this into a kernel module.

Re:Performance

[Ulrich Hobelmann]

Well, you can bet your ass that Windows's native NTFS is much faster than the Linux one, because they wrote the FS, and they have years of time to optimize the working driver.

Sure, user-mode will be a performance issue, but I think the context-switch + work is only necessary, when the kernel decides to either read data (on a cache miss) or write out its file buffers to disk. So if you don't use NTFS as your native disk (not sure how that'll work with permissions and stuff), you should really by fine. Maybe there's an option somewhere to turn on big read-write chunks (so that the kernel will always read/write huge blocks from the user daemon, which would make context-switch time negligible).

Re:Performance

[jrcamp]

Why would multiple users be using it at a time? The main use case for NTFS is recovery and people who need access to their files on dual boot laptops and desktops.

Re:Performance

[Reality Master 101]

Any sources or references for that?

Performance problems are a well-known fundamental problem with microkernel architectures that use user-mode processes. If you're interested in the subject, there are lots (and lots) of discussions about it (hint: your instincts above are wrong). Google is your friend.

Re:Performance

[zataang]

I fail to understand your logic. Why would you have multiple people accessing a Windows partition? Either you have a Windows server, or you have a Linux server. I just don't foresee having an NTFS partition on a Linux server (which serves multiple users) being so actively used as to cause performance problems. Also, being in the user space has its own advantages in terms of robustness. Given that ntfs is not documented (well?), I would be much more assured if the module stays out of the kernel. Why do I need to suffer random crashes?

Re:Performance

[kevin_conaway]

Why would users even need more than 640k of ram?

Re:Performance

[iamacat]

Actually, they should do no such thing. Kernel is big and complex enough as it is. Kernel drivers can not use virtual memory, explicit multithreading or advanced algorithms from MP, STL and hundreds of other C++ libraries. And when they have bugs, the system really crashes and burns as opposed to just a couple of processes getting file errors.

Re:Performance

[Schraegstrichpunkt]

Performance problems are a well-known fundamental problem with microkernel architectures that use user-mode processes.

It's a widely-believed myth, mainly due to the poor performance of bloated first-generation microkernels like Mach, although I suppose it probably also applies to Linux when Linux acts as a microkernel.

Google is your friend.

Just because Linus Torvalds thought something was impossible during the 1990s doesn't make it so, so I suggest you skip the infamous Linus vs. AST discussion from that time period.

The reality is that:

1. Microkernel architectures are hard to design. This is suspected as being the real reason why they are not very popular today.
2. Monolithic kernel architectures are prone to insecurities. There is just way too much privileged code, and too many failure scenarios.

Unlike Linus, some people are actually devoting much of their time to solving these problems. AST is one such person. See this page on the subject.

Re:Performance

[ArbitraryConstant]

While an NTFS kernel module with awesome performance would be nice, we haven't even had reliable writing before this (not without the Windows NTFS driver). It wasn't just a lack of performance, it was so bad that you'd need to do stuff like keep an FAT32 partition for transferring files. You won't run your high-traffic website on this, but it is still extremely helpful.

For the purposes of making a dual-boot system less painful, it's great. Now all we need is a Windows driver for Reiser...

Re:Performance

[Reality Master 101]

It's a widely-believed myth, mainly due to the poor performance of bloated first-generation microkernels

No, it's a widely-proven fact. Now, it may be possible to have a decent performing microkernel, but to dismiss performance problems as a "myth" is disengenious at best.

some people are actually devoting much of their time to solving these problems.

Exactly. "solving", implying that performance problems are hardly a solved issue. Now, I'm not here to knock microkernels -- they certainly have their place, and the advantages may end up outweighing the performance problems (as high level languages did with assembly language), but microkernel advocates really need to get over their oversensitivity on the issue. Screaming "myth" when everone's experience is to the contrary does your movement no good.

Advocates ought to be saying, "Yes, microkernels are slower than monolithic kernels and probably always will be due to their nature, but with care the cost can be minimized, and we believe the clear benefits outweight the cost.

That's at least a debatable statement.

Re:Performance

[azhrei_fje]

Kernel drivers can not use virtual memory, explicit multithreading or advanced algorithms from MP, STL and hundreds of other C++ libraries.

Wow, I don't know where to start with this. First, mod parent down using -1, vague.

Have you written a filesystem? I have. The "virtual memory" you're apparently referring to is the process' virtual memory. This sounds like what you're trying to say is that the protection provided by process virtual memory isn't available in the kernel. And that's true. But that doesn't change the fact that the kernel can map any physical memory to any effective address. So the statement that the "kernel can not use virtual memory," is extremely bogus. ("Bogus" is a technical term. It means "completely wrong.")

Concerning "explicit multithreading," you must be referring to the idea that the kernel can't call pthread_create()? But there's no reason for the kernel to do so. Multi-threading in a user-level process is often done to achieve concurrency related to the delays that a single-threaded application would have if it had to wait for I/O to complete. The kernel doesn't have to do that at all. The kernel would queue up the I/O request, then continue on its merry way. When the interrupt from the device signals that the I/O has completed, a separate handler takes care of it. In a user process, threads are useful in order to modularize the I/O functions. In the kernel, they often aren't needed, since callbacks are used instead. Same functionality, different design technique. And even if they are needed for some obscure reason, all modern kernels (Linux included) support kernel threads. (My SUSE Linux 10 box currently has 19 kernel threads executing.)

The "advanced algorithms" you're referring to are probably coming out of user-space libraries. And in this regard, you're correct -- user-space libraries cannot (currently) be linked into the kernel and there is plenty of debate about whether such abilities should even be attempted. (The problem with user-space libraries inside kernel space revolves mostly around bugs and implementation deficiencies. The truth is that an algorithm that is mostly cpu-intensive probably could be loaded into kernel space using some kind of hack, and there are open source projects that are already working along these lines.)

In any case, there's no reason why those algorithms couldn't be executed inside the kernel. For example, take the find() generic algorithm from the STL (a macro from one of the libraries you mentioned). Why can't I use it? (The truth is I can.) And why can't I use the list class from the same library? I admit that linking large objects into the kernel could result in quite a bit of bloat, but there is not a technical reason that it couldn't be done. (Except in the case of C++ exceptions within the kernel. There is a group that has patches available for the Linux kernel that add support for C++ exception handling. With those patches, any STL code should be able to work in kernel space, although I've not tried it personally.)

It seems to me like the parent has read a magazine article and jumped to conclusions. Or perhaps they are even an experienced developer, but took huge liberties with the wording of their statement. But as "Captain Obvious," I felt it was my slashdot civic duty to clarify he issue. :)

Re:Performance

[quintesse]

#1 : To which the same AST says "The complexity is quite manageable. This is not speculation. We have already implemented the system, after all."

He says that are basically only a few basic classes of drivers that you need this kind of complexity for, I imagine extending one for your particular filesystem, hardware, etc is a lot easier than having to write one from scratch each time.

Re:Performance

[quintesse]

but to dismiss performance problems as a "myth" is disengenious at best.

come on! The GP clearly said:

Performance problems are a well-known fundamental problem with microkernel architectures that use user-mode processes

Look at the keyword "fundamental" here, THAT's the myth and the fact that several people, AST being one of them, have proven that there is no such "fundamental" difference is the "fact" here. Even coming up with a long list of examples where the performance is actually worse does not mean a thing, it might just mean it's very difficult or that no microkernel has ever gotten the same kind of attention as monolithic kernels.

Yes, microkernels are slower than monolithic kernels and probably always will be due to their nature

Go read some of the work being done on microkernels, jeez just reading http://en.wikipedia.org/wiki/L4_microkernel_family will give you enough leads, there are _actual_ _working_ examples of microkernels that perform well, in some cases outperforming existing monolithic kernels.

That's why people are screaming "myth" because people like you have already made up their mind and keep repeating the same old adage when it just isn't true.

Re:Performance

[Reality Master 101]

Look at the keyword "fundamental" here, THAT's the myth and the fact that several people, AST being one of them, have proven that there is no such "fundamental" difference is the "fact" here.

AST himself said at the site the poster above linked to, "In this paper we argue that for most computer users, reliability is more important than performance and discuss four current research projects striving to improve operating system reliability."

If performance is exactly the same or better than monolithic kernels, as you claim, why would AST make an issue that reliability is more important than it, unless performance WAS an issue? Why wouldn't he write a paper titled, "Having your cake and eating it too... better performance AND better reliability. Why microkernels have won the war."

The answer is because they AREN'T and NEVER will be for general purposes. Sure, you can find isolated tests or isolated projects where they might do better (and the cost of doing better is generally insane complexity), but it's just foolish to argue that they're anywhere close in performance in the general case.

Look, extraordinary claims require extraordinary evidence. Show me the big database servers running on microkernels. Show me the big web servers. Show me big mail servers. And show me how the performance compares to the monolithic kernel operating system on the same hardware.

Sure, microkernels "work", but who cares? I can get DOS to "work". Show me something that works *better*. Or to put it another way, when microkernels are truly better, you won't need to sell everyone, they'll sell themselves. So far, they haven't for general purpose operating systems that care about performance.

Not Just Linux

[TheRaven64]

FUSE has been ported to FreeBSD, and it appears that the driver also works there.

Re:Not Just Linux

[foniksonik]

Which means that this may make it into OS X 10.5 ;-p Hurray! Which potentially means that we Mac users 'could' run Windows dual boot AND use Parallels/VMWare with the same install of Windows AFAIK.... ???

Re:Not Just Linux

[Tim Browse]

I changed the motherboard in my Windows machine once, but didn't reinstall the OS straight away - was curious if it would 'just work'.

I was running a dual boot Windows 98 and Windows 2000 system.

Windows 98 booted up and said "Crikey chief! It's all changed!", then had a bit of a scurry, and rebooted.

Then it did it again.

And again. And again. And again.

Then it worked.

After 5 reboots all my hardware had been auto-detected and configured, and Windows 98 was ready to go. I never had any problems with the installation after that.

Then I booted to Windows 2000. It crashed out in the text mode boot screens and died in a flaming heap, telling me I'd committed some heinous act or other. I never got it past that. In the end I did a clean install of Windows 2000.

I was impressed with 98, not so much with 2000. Surprising, as I'd fully expected the opposite result.

Re:Not Just Linux

[ecliptik]

He's right, it's an issue with the IDE Drivers. Before you swap motherboards on a Win2k+ install make sure that the IDE driver in device manager is set to generic. Otherwise you'll get that dreaded STOP Error.

Here's a good link for it:

http://www.windowsreinstall.com/install/other/moth erboard/problems.htm

Re:Not Just Linux

[mczak]

That "heinous act" it told you must have been "inaccessible boot device".
For W2k and XP (not sure about NT) there are basically 2 drivers which it absolutely needs to be able to boot. One is the graphic driver, which has fortunately a fallback to generic VGA. The other is the disk driver, so if you change your board to one with a different hd controller (for typical setups that means different chipset, though you might get lucky if it's a different chipset but which can use the same driver, dunno) then it will not boot. There are documented ways around this (for instance in the MS knowledgebase), though yes if you ask me it's really lame that there is no generic ide fallback. Apart from that board swaps seem to work pretty well with W2k (even though not recommended by MS), of course you might need to change hal (uniprocessor to multiprocessor and such things) and other drivers later.
(Actually IME hd cloning / swapping is more problematic due to recognition of drives with some unique identifiers)

Re:Not Just Linux

[algae]

if you ask me it's really lame that there is no generic ide fallback.

There is. Go to the device manager and find your IDE controller. Click "update driver" and then manually choose a driver from the list. Select "Generic IDE DEvice" or whatever they call it. After this driver is installed, you can reboot the machine with a completely different motherboard.

No 64-bit

[Bill Dimm]

Looks like a great piece of work. One important note from the article:
Problem: Why doesn't the driver work on 64-bit and bigendian systems?
Answer: We have no resource for that. Neither hardware, nor workforce.
Status: Low priority.

Re:No 64-bit

[Tim Browse]

surely any well-written code for a portable operating system, such as Linux, should be written in such a way that endianness and word size is irrelevant?

I would have thought that fiddling with bitwise data structures on disk is pretty much one of the cases where endianness and word size is very relevant. I mean, I could be wrong.

In fact, these days you'd pretty much have to go out of your way to ensure that it didn't work.

Hmm. Let's try reading a 32-bit value from the disk surface and masking out the top 3 bits (random file-systemesque example chosen off the top of my head). Yep, I reckon you'd have to go out of your way to make sure that it did work across platforms with different word size or endianness. It's not exactly hard, but you'd certainly have to go out of your way.

Especially as I'm guessing the number of times this driver has been run on a Linux PC that is not a 32-bit Intel x86 machine is currently close to zero, given the typical application of this software.

Re:No 64-bit

[david.given]

Let's try reading a 32-bit value from the disk surface and masking out the top 3 bits (random file-systemesque example chosen off the top of my head).

uint32_t read32bit(off_t offset)
{
uint8_t buffer[4];
fread(buffer, 4, 1, disk);
return buffer[0] | (buffer[1]<<8) | (buffer[2]<<16) | buffer[3]<<24);
}

{
uint32_t value = read32bit(offset) & 0x1FFFFFFF;
}

Totally portable, totally trivial, and efficient --- any reasonable compiler will optimise out the 'return' line if it can.. (I'm assuming little-endian. May contain typos.)

This is basic programming skills. There is nothing the least bit hard here, and these days, people should be doing this kind of thing as a matter of course.

Re:No 64-bit

[Briareos]

Problem: Why doesn't the driver work on 64-bit and bigendian systems?
Answer: We have no resource for that. Neither hardware, nor workforce.

I suggest someone get them an NSLU2 from Linksys - they retail for 80EUR here and have a 266MHz ARM CPU that can run in either little or big endian mode, with several distributions of Linux already available to install on them...

np: Duo505 - Facing It (Live) (Monsters Of Morr Music)

Re:No 64-bit

[Tim Browse]

I refer you to your original comment:

"In fact, these days you'd pretty much have to go out of your way to ensure that it didn't work."

and then your supplied code:

return buffer[0] | (buffer[1]<<8) | (buffer[2]<<16) | buffer[3]<<24);

That code looks like it's going out of its way to deal with endian issues to me.

As I said, it's not hard, but it's not free either. Perhaps we should just agree that your original comment was hyperbole and leave it at that.

Re:No 64-bit

[waveclaw]

Problem: Why doesn't the driver work on 64-bit and bigendian systems?

I can confirm that it compiles, installs and works on 64-bit SuSE Linux 10.1 (AMD Athlon(tm) 64 Processor 3000+.) It does require you to get the latest 2.5 fuse (http://fuse.sourceforge.net/). And I do have a complete 32-bit environment installed, but the ./configure was clearly x86_64.

# arch
x86_64
# uname -r
2.6.16.13-4-default
#lsmod | grep fuse
fuse 39192 2
# /usr/local/bin/ntfs-3g /dev/hda1 /windows/C -o rw,user
# cd /windows/C
# ls
# touch autoexec.bat
# ls
autoexec.bat


And I am just one checkinstall away from an rpm. (Too lazy to write yet another custom SPEC file today.)

Re:Yay

[Anonymous Coward]

What's the last closed-source file system you completely reverse-engineered, then?

Awesome!

[The MAZZTer]

Except that every partition tool under the sun fails on my NTFS drives, so I can't even install Linux anywhere... so my computer is STILL Linux proof.

Re:Awesome!

[lord_rob the only on]

A reply from the developper :

Currently I'm not interested in the kernel driver. It's a lost case for over a decade. Full read-write could be done in user space pretty fast and I can't see drawbacks, only benefits:

- NTFS is huge and complex, not for the kernel. Crash in kernel (hw error, corrupt ntfs, etc) and game is over. Crash in user space then just restart the service.
- kernel has a lot of limitations, restrictions which are all gone.
- fedora/redhat users have never ending hassles with installing the driver. Instead they could install ntfs-3g once and forget the issue forever.

Re:Awesome!

[rthille]

Try booting a live linux CD, then using 'dd' to zero out the first 16K of the disk.
Then reboot from the CD and reformat the drive.

Sometimes the partition software can get confused by what's there, and the kernel will cobble up reasonable information about the drive from the drive's response itself.

Of course if you want to preserve your NTFS partition that's not a good approach. However, I've had bad luck with resizing windows partitions, so my approach is to backup, reformat, reinstall, and reload from the backup.

Re:Awesome!

[ABoerma]

SUSE's intaller YaST let me resize NTFS partitions. You might want to try that.

Re:Awesome!

[Bill Dimm]

Have you tried the latest GParted? This article says a new version was released on July 9.

Re:Awesome!

[ratboy666]

How to do it...

Assuming partition tables are "fubar".

#1 BACK UP ALL YOUR DATA. This is normally a sign of a failing drive.

#2 Download and burn a bootable CD of you hard drive vendors diagnostic kit.

#3 Run it, and "recertify" your drive. May take a couple of hours (and, you may just want to dumpster the drive, if your time is valuable). If the drive does not certify, discard it.

#4 Boot your system with Knoppix, or another recovery Linux system. Issue the command: "dd if=/dev/zero of=/dev/hda" (replace hda with hdb, hdc, hdd, etc. depending on which hard drive it is).

#5 Run Linux partitioning tool "fdisk /dev/hda". You *will* get a complaint about an improper partition table, which is ok. Partition, and write the new partition table.

#5b Alternatively, boot a Linux installation CD, and load Linux. Ignore warnings about "improper partitioning", and choose to have the partition table replaced.

The IMPORTANT steps are 1 to 3. If the partition table cannot be manipulated, it is an almost sure sign your drive is heading south.

Ratboy

Re:Awesome!

[OmnipotentEntity]

Linux *can* have ntfs support in the kernel. The developers just chose not to do it that way. FAT32 support is very stable. NTFS is still in flux as Microsoft is still developing it.

From TFA, The driver currently is in BETA status: before release of this software we haven't experienced any driver crashes or data loss during our heavy quality testing, however we are aware of some minor issues which will be resolved in the near future.

It's still in beta. While it's a GPL'd beta, and probably far more stable than beta implies. It's still beta, and you should expect it to bug out from time to time (whether or not this is actually the case.)

So, if you have it when it bugs out as a kernel module you get a kernel panic. I don't like kernel panics, and you shouldn't like them either. So keeping it to the userland is probably best for now. Do you really need it as a kernel module?

Re:Awesome!

[lord_rob the only on]

But Linux has ext[2|3], reiser[3|4], XFS, JFS, ... in its kernel, why can't Windows ?

Re:Awesome!

[LinuxGeek]

Help is all around!

Re:Awesome!

[Dave2 Wickham]

Actually, it's not quite that simple; Windows (2000, at least) doesn't like it if it's not on the primary HDD. You have to make Windows think it is on the primary HDD to allow it to boot. In GRUB it's a case of adding the following lines:

map (hd0) (hd1)
map (hd1) (hd0)

Not a major change, but without knowing this beforehand, you can't just get Windows to boot off a secondary HDD.

Time to change my main partition...

[the_humeister]

...from usmsdos to ntfs! Finally!

Re:Yay

[Anonymous Coward]

Still years before NTFS will be documented. And Microsoft doesn't supply a ReiserFS or Ext3 driver even though those filesystems are documented.

WRONG - ntfs-3g != ntsfmount

[Chris Pimlott]

This new driver is "ntfs-3g". "ntfsmount" is the previous NTFS driver that ntfs-3g is based on. Since ntfs-3g is brand new, most of the documentation on the Linux-NTFS site is about the older driver. ntfs-3g promises practically unlimited file creation and deletion.

Cue drumroll for New! Improved! MS file system

[dpbsmith]

This is the cue for Microsoft to roll out a new! improved! disk directory format.

If I were Microsoft, I'd make just enough undocumented changes to screw up reverse-engineered implementations of NTFS... providing just enough increased functionality to which I could Point With Pride.

I might even called it WinFS 2007 or WinFS X-Treme or Enterprise WinFS. It wouldn't have anything to do with the real WinFS... anything more than Javascript had to do with Java, or Mac OS 9 had to do with Copland, but it would certainly muddy the buzzword waters.

Imagine a meeting with nerds and suits present in which the nerds make the mistake of mentioning Microsoft's failure to deliver WinFS, the suits would wave their magazine and say they had, then drum their fingers, yawn, and look at their watches while the poor nerds try to explain the complex technical issues and how WinFS was supposed to therblig the frammistan while WinFS Gold merely globulorns the ferthbernder.

Re:Cue drumroll for New! Improved! MS file system

[Anonymous Coward]

Two points:

1. The likelihood that anyone at Microsoft cares about whether or not Linux can read and write to NTFS is vanishingly small. If anything, a good NTFS driver for Linux means more people who dual-boot will use NTFS for their shared partitions, which might be marginally good for Microsoft.

2. From what I've read, WinFS (which stands for Windows Future Storage, not Windows File System) isn't a new file system at all, just a set of database services that run on top of NTFS. That's why Microsoft were able to delay it until after the Windows Vista release, since it can always be added to a Windows Vista system later on, without any changes to the underlying (NTFS) file system.

Re:Cue drumroll for New! Improved! MS file system

[schmiddy]

This is the cue for Microsoft to roll out a new! improved! disk directory format.

I think it's great that Linux has gotten this far with NTFS reading/writing reverse engineering. I've used the shaky NTFS support for quite a while. One key use is when you forget the Admin password on Windows, and you're either locked out of the system or have only user-level privileges, you can use a Linux bootdisk the open the (otherwise hidden) password file and blank it out. The NTFS drivers have always had dire warnings about writing to NTFS possibly resulting in corruption, so this step forward is great.

However, I think you're exactly right – one way or another, MS will find a way to tweak the filesystem, probably in Vista, maybe even as an auto-updated "security fix" to XP, that will make compatibility even harder. Look at how long it's taken us to get this far. Linux has been working on NTFS support since.. at least Win2k, which is a looong time. And now we have to worry about MS putting us back to square one.

I think it's a little sad that Linux has to waste so much time being compatible with MS software (.doc support, NTFS support, FAT support, samba). I'm not in any way saying we shouldn't be doing this, since everyone wants compatibility, but it's a real PITA we have to waste so much time playing catch-up with MS. Imagine where OpenOffice et al. would be today if they didn't have to worry about reverse engineering the miserable .doc format. We could wish that MS would play nice and ues open standards, but there's a snowball's chance in hell of that happening. They know exactly what they're doing, and how much it sets us back. Perhaps someday MS will be forced to play catch-up with us for a change. I guess you could say the IE team is doing that right now, but it's not like we're implementing non-standard features in our browser.

Re:Cue drumroll for New! Improved! MS file system

[evilviper]

If I were Microsoft, I'd make just enough undocumented changes to screw up reverse-engineered implementations of NTFS... providing just enough increased functionality to which I could Point With Pride.

And they'll call it NTFS6...

They already did something like this with NTFS5, found in Windows 2000. Once Windows 2000 has booted-up with access to an NTFS partition, you can't run chkdsk from NT4 (or older) anymore. You're actually stuck with 2000, whether you like it or not. God help you if you remove your Windows 2000 installation.

Personally, my hopes lie with FFS Drv. Absolutely EVERY major operating system other than Windows can read/write to UFS/FFS.

Re:This is great news

[jZnat]

When I installed Vista Beta 2 on a friend's box, it required use of NTFS for its root partition (the C: drive as you Windows people like to call it), so I'd say that Vista will use NTFS. Then again, Vista might as well use Reiser5 or something, because that will be the default filesystem on Debian Stable kernels by the time Vista comes out.

Re:This is great news

[bogaboga]

The `beef' is not whether Vista will use NTFS. The concern is that even if it uses NTFS, there will be an update to this NTFS that will "force" Linux into catch-up position. Guys, this develpoment is not in Microsoft's interest. Remember what happenning to Samba with CIFS/SMB? The underlying protocol remained basically intact but Microsoft kept updating the CIFS/SMB protocol.

Result? We were and are still playing catch up depending on who you speak to.

Is it stable?

[rsilvergun]

as I recall, the problem with the kernel driver is it's not considered safe for writing. There's plenty of antecdotal evidence that it's ok to write, and I've done it, but has the been run through it's paces?

ntfsmount != ntfs-3g

[Chris Pimlott]

ntfs-3g is brand new and it not the same thing as ntfsmount, which is what the current documentation covers. Please read the ntfs-3g announcement, which promises practically unlimited file creation and deletion.

It is good news ... But ...

[vtcodger]

Full NTFS compatibility in Linux is a good thing. There are a gazillion scenarios where it is necessary for users to get at Windows files from Linux or vice versa without moving stuff over a network.

But, keep in mind that NTFS remains proprietary and Microsoft can break it for newly written files any time it suits their business purposes to do so. All it takes is one update.

No one but me seems to care about this, but I think that the proprietary and undocumented nature of NTFS is an important reason why System Administrators need to have a workable exit strategy for Windows. They don't need to exit now. But in three or five or ten years if (when) Microsoft decides to lock in its user base, users should want to make sure that they have the option of being outside the door that Microsoft is slamming shut.

Re:It is good news ... But ...

[Martin Blank]

But, keep in mind that NTFS remains proprietary and Microsoft can break it for newly written files any time it suits their business purposes to do so. All it takes is one update.

Hypothetically, yes. However, there are few things that they -- or any OS developer -- are more paranoid about altering than the filesystem. You can recover from a bad driver, or a bad patch for most functions; recovering fully from a bad filesystem alteration may be nigh-on impossible, and Microsoft is going to think really, really hard before they go and do something that may result in lost data.

Re:It is good news ... But ...

[peragrin]

On one hand I agree with you. On the other MSFt is the same Company that brought us Fat32, Win Me, and Active X.

Because every web developer in the world needs direct write access to any where they like on the hard drive, that doesn't have permission controls of any kind.

Re:It is good news ... But ...

[puffing_billy69]

The entire reason Microsoft is the most successful business in the world is that they stay compatible with more stuff for longer than anyone else.

Nonsense: NTFS is a Windows Filesystem. It's never had to be compatible with anything else, and they've clearly made no effort in letting it be. They have made changes in the past that have had consquences. (I'm not saying they made those changes to be evil, the point is, XP broke Nortons).

And who said they'd need to break XP NTFS to break Linux NTFS? I have no doubt at all the current version of NTFS is already prepared for whatever other layers of obfuscation they might add. Mark my words, NTFS was designed only to allow Windows to store and retrieve files, not you. Windows will be your vehicle for storing and retrieveing those files, they've worked damned hard at making it that way, and they'll continue to work damned hard.

Re:It is good news ... But ...

[Antique Geekmeister]

No, Microsoft deliberately breaks compatibility. I suggest you look at the old FreeDOS lawsuits, and the Netscape lawsuits where Microsoft tried to pretend that NT Workstation couldn't work as a Netscape server, the oddness of Internet Explorer and their refusal to make it removable, and the weirdness they did to Kerberos that MIT sued them successfully over, and even take a look at what Active Directory does to DNS, and the current lawsuits about ODF in Massachusetts, and the recent EU lawsuit whaere Microsoft refused to document their software, instead publishing deliberately obfuscated and unusable documentation.

Re:It is good news ... But ...

[Martin Blank]

If they did release an update that trashed 1 or 2 % or Windows installs, they'd release another patch and the public would keep lapping it up.

Take off the FUD hat and re-read what you wrote there. Do you really think that trashing 1%-2% of the install base would be smoothed over with a patch?

Back in 2004, when SP2 was released, the estimated installation base for XP was more than 250 million systems. Throw in Windows 2000, plus growth since then, and I would think that a conservative estimate of 400 million 2000/XP/2003 systems wouldn't be out of line. The suggestion that four to eight million systems -- servers included -- could lose data and that Microsoft could just release a few new bytes to handle it is ludicrous at best. Microsoft would be hit with a class-action suit -- and deserve it -- for taking out so many systems.

You can break drivers. You can break compatibility. You can even break mainboard hardware. In each case, people will move on. But when you break people's data, forgiveness is often impossible to find.

EVEN BETTER NEWS

[dsginter]

Think of the implications:

A given distro can now come with a handy Windows InstallShield Wizard and INSTALL UNDER WINDOWS and BOOT/SHARE the same partition.

This is huge. Who wants to be the first to make a Linux ActiveX malware distro?

We always could write to NTFS

[r00t]

dd if=/dev/zero of=/dev/hda1

It's really fast, despite being in userspace, though it can still take a while because there is so much that it needs to do. Start it before you leave work, or before you go to bed.

As a side effect, your NTFS partition will finally be free of spyware. It's the only way.

Re:We always could write to NTFS

[RogL]

I say we dust off, ssh in and dd if=/dev/zero of=/dev/hda1 it from orbit. It's the only way to be sure.

Re:Why?

[SirTalon42]

1.) WinFS was canned a good while ago
2.) Despite its name, WinFS is NOT a new file system, just a layer on top of NTFS.

Re:My thoughts on this...

[tomhudson]

Too bad he wasn't more honest and said "just STEAL your ideas from everyone else and call it a Unix operating system" You're all nothing but fucking thieves!

Well, you've been stealing ideas about communication - you've stolen most of the alphbet, some punctuation, even some html formatting codes, to create your post. Couldn't come up with something original? To use your own words, "You're nothing but a fucking thief!"

(actually, you're JAW - Just Another Wanker)

Re:My thoughts on this...

[mh101]

Um, since when is 'interoperability' the same as 'lack of innovation' and 'stealing'? Nobody's trying to 'steal' NTFS to use in Linux. Rather, people are looking for a way to access their data from Windows that's stored on an NTFS partition. I don't think any Linux users would willingly give up EXT3 or ReiserFS for NTFS.

Re:This is great news

[Schraegstrichpunkt]

One nice thing is that Microsoft can't change things willy-nilly with NTFS as it could with, for example, the Word file format. The worst problem with NTFS write support is that a naive driver can cause data corruption. Once the free/open-source driver is sophisticated enough, there won't be much Microsoft will be able to do to exclude it, except by adding new optional features. There will come a point where anything that Microsoft does to break the free driver will also break older versions of its own drivers. Microsoft can't really afford to let that happen, since once thing businesses will not tolerate is a file system that arbitrarily loses data, especially since NTFS is currently viewed as being very stable in the Windows-using world.

Breaking filesystems is much more drastic than breaking network protocols. The only thing that Microsoft could do that would effectively deter users of the free driver is to make it (and any older version of Microsoft's own NTFS drivers) cause data corruption. Even Microsoft isn't stupid enough to do that.

That's a pity

[Darkforge]

... because there is a really good reason to support NTFS in the kernel: so you can boot off of an NTFS drive. That would eliminate the need for Windows users to re-partition their drives when installing Linux, and allow for an easier dual-boot.

Re:That's a pity

[bfree]

You could use this to boot from an ntfs partition if you wanted, just use a kernel based read-only driver to get to an initial ramdisk which includes enough support to remount the partition with this. Even without this you should be able to use a loopback filesystem on ntfs even with the older ntfs drivers as their write support is at least meant to be solid if you are not creating or changing the size of files.

Re: My stealing of this...

[Jesus_666]

That's because we're all evil thieves (chaotic evil, actually). Yup, Linus Torvalds is a level 15 Thief/Wizard specializing on Thievery:Copyright Infringement. After we have stolen Windows we will proceed to steal Christmas, the fire and your time. (Then again, we already are doing that last part, via this very site.) But we aren't done yet. After we're done stealing Microsoft we will steal Bill Gates' thoughts from his mind and sell them on the black market. Which we will also steal.
Now excuse me, I have to go to my (obviously stolen) kitchen and steal me a sandwich. The flavor is peanut butter crime.

Re:My thoughts on this...

[Jairun]

You are a moron.... I don't know any linux user who would willingly use NTFS. We are only looking to access files from the hated windows partition. Not to mention, sometimes it's easier to fix windoze when you are not acctually running it.

use ext3 in windows instead

[radarsat1]

I found I prefer simply being able to access my Linux partition from Windows by installing the (unfortunately not open-source) Ext3 driver.

Seems to work quite well.
Yes, unfortunately it can't be Windows' root partition, but at least I can use Windows & Linux without needing an EXTRA data partition, or using Windows on FAT32.
(Though I usually do just use FAT32 to keep things easy, because I'm not all that worried about security on my home box.)

Anyways one problem I ran into using a shared FAT32 partition is that I couldn't use files > 4GB. Haven't seriously tested it yet but I think using the Ext3 driver will fix that. (Mainly for virtual machine images for Qemu.)

Steve

Re:use ext3 in windows instead

[cortana]

FYI, ext2fsd is a Free Software alternative to the fs-driver.org IFS.

Does a good job - needs more testing and funding.

[wmaster]

I have successfully installed and tested it on most recent Debian Sid/Kanotix with kernel 2.6.17 - http://kanotix.com/. Creating and deleting folders (even in root directory), adding more than 100 files to a folder and deleting them again, removing some 100 temporary files, copying a 1,7Gb sized iso-file and moving it around - all that was possible whithout any error. A very promising initiative from a developer to get things moving again in this mine field of myths. As he is a true open mind he contacted first the existing ntfsprogs-project and handed all his work over to them - just a pitty that the head developer there recently started to work for Apple, and announced that he is not interested in getting a solution for Linux out, before he finished the same for his new employer next year. I would be more than happy to support the ntfs-3g developer getting his own project running, and also finding sponsors in order to solve this nasty hardware problem. Anybody interested in helping? ;-)

Personally never understood why people always make

[msimm]

this same argument. I'm glad for NTFS support myself, but a Linux recovery disk is definitely not the best solution for actual systems recovery. And were did we all seem to get this idea that Linux bootable disk were the only bootable disks anyway?

I'd suggest taking a good long look at UBCD4WIN. Its *is* a bootable disk. It runs the Windows kernel of your choice (you build it off your own disk, but the process is much less painful then it sounds). It also happens to include a slew of native Windows programs/utilities for doing things like...password blanking, virus/spyware detection/recovery, partition recovery/disk repair, Windows networking, including SMB access for recoveryies where you can't get the core functioning but still need to retrieve those files.

It is an all around good project and I'm sure I'm not even remotely doing it justice. Of course best of all, its native NTFS (assuming you build XP or a variety that supports it) so you don't have to worry about write problems in the same way.

I work as a systems admin at a mainly Linux shop so I don't get much cause to use it, but its something I'd never leave home without. I'm sure I've got a Knoppix disk sitting around somewhere, but for (Windows) system repair there's simply no advantage.

I sound like a commercial. :) Donate some money or something if you find it useful. Its free after all, but the guys time can't be.

FUSE is too slow

[caseih]

The latest knoppix CD uses an older version of this NTFS driver (read-only if I'm not mistaken) via FUSE and it is *slow*. Rsyncing an entire disk for backup purposes can take days (yes days). Disabling the fuse-ntfs system in knoppix and mounting using the read-only NTFS kernel-level driver is several orders of magnitude faster. So I think this driver is good for sharing data and doing emergency stuff, but it is no where near fast enough to think about using it as a root file system or anything. Knowing this latest driver is faster than Paragon's driver is good news; paragon's driver must have been even slower.

When the ntfs driver is stable, I hope it will be put in the kernel (at least as a native file system). Then we can consider adding a unix layer on it and install linux to the same drive as Windows, for those that want to dual-boot.

Now it's time to...

[ivan256]

In the grand tradition of open source NTFS drivers, this project has now reached the point in it's lifecycle where the developers abandon it and all future implementations start from scratch.

I went that route as well.

[HoneyBunchesOfGoats]

I use a large ext3 partition to store video files, as I'm experimenting with video editing in both Linux and Windows. No hiccups so far using the ext3 filesystem for big video files in Windows. One thing to be aware of, though, is that Windows sees it as ext2, so you lose the benefits of the journalling filesystem there. I haven't lost any data yet, but it is something keep in mind.

Re:Still using FAT32

[Anonymous Cowled]

DOS natively supports fat32, but does not natively support NTFS - tha's what NTFS4DOS is for (google it). Incidentally, for Windows, NTFS is a far superior fs than FAT...

Ubuntu Install

[Aladrin]

If you want to install this, you'll need FUSE 2.5. (K|X)Ubuntu only has 2.4, so you'll need to get an update.

http://www.debuntu.org/2006/06/26/71-fuse-253-for- ubuntu-dapper/

I've installed that on my desktop machine and managed to mount my ntfs drive (for dual boot) and read files. I didn't try to write anything yet, though. It seems to work fine.

Enjoy!

Re:Yay

[julesh]

Random aside:

NTFS was actually launched in 1993, 13 years ago, when Windows NT 3.1 (really 1.0, but the version was matched to the MS-DOS-based Windows 3.1) was released.

It's interesting to note that this means XP (which identifies itself internally as NT 5.1) is actually NT release 3.1.

3.1 is typically the best version of any microsoft product (except DOS; 3.3 was generally regarded as better). Version 4 (e.g. Win95, DOS 4.00, ...) is often a complete flop, frequently requiring a quick followup release (W95OSR2, DOS 4.01) to rectify serious problems with it. At this point consumers start to lose cofidence and MS look for a new direction in order to convince people that their software isn't all that bad.

So, when Vista flops, what are MS going to replace it with?

Just tried it out...

[pxc]

And it works fine. The only problem is that SuSe 10.1 doesn't load the fuse module before it tries to mount the fstab, so after I boot I have to do a "modprobe fuse" and "sudo mount -a" to get it to work...

PS: I've tried both ntfsmount and captive-ntfs on this same system and neither have worked.