Slashdot Mirror
Tom's Hardware Looks At WinFS
Alizarin Erythrosin writes "Tom's Hardware Guide has an article about the new WinFS file system. The article talks first about some of the problems and advantages with FAT[16|32] and NTFS, then talks briefly about WinFS. Here is the summary: 'Microsoft is breaking new ground with Longhorn, successor to XP. The upcoming WinFS file system will be the first to be context-dependent, and promises to make long search times and wasted memory a thing of the past. Today, THG compares it to FAT and NTFS.' Personally, I still have reservations about using a relational database to keep track of files. Unless they can keep the overhead to a minimum, I can't see it being as efficient as a file system should be."
i wonder about the performance for increasingly large storage space...
/. consensus is journaling = good...
do you think that a search taking O(log n) [this is assuming similar performance to oracle] time over exponentially large drive space is better in the long run than what we've got now? is hfs+ that bad taking care of my 40gb drive?
maybe i'm being less open minded about the journaling, especially since the
and i am running a g4 400 w/ macosx and i really would hate to see that 10 percent get sucked up by disk accesses.
----
http://www.hellection.com
Hear hear! I paged through the whole thing hoping to find some _actual_ information, but was really disappointed on page 6 which was basically the summary over again.
AmigaDOS had an 80 character or so comment field for each file. The comment didn't show up when you did a DIR, but when you did a LIST it was there. All of my pictures and sounds had nice descriptions in the comment, size/depth/mode/etc.
Very useful.
This article is tripe.
The closest it gets to examining the (possible!) new Windows filesystem is calling it a relational database, and going on for a bit about how paths (ie: directory structures) will be irrelevant. Oh, and yeah, the closest thing he found to the implementation was called "winfs.exe" and did nothing but produce errors.
The bulk of the article is a (poor) attempt at explaining filesystems in general, and FAT and NTFS in particular. However, it gets a number of things wrong and - at best - garbles a lot of things. If you already know what he's trying to say, you *may* be able to pick out truths, but if you don't you'll walk away with misinformation.
I would suggest instead perusing arstechnica.com and aceshardware.com. I don't know if they've done any filesystem stuff, but if they have it'll be of reasonable quality.
Barclay family motto:
Aut agere aut mori.
(Either action or death.)
It's really irritating to have a RAID that gets crippled to 5-10MB/sec when on other OS and FS combos it can do 80-100 because MS has decided "Oh performance isn't important, reliability is, so we'll force cache off for all SCSI miniport devices even if it says it's on."
f =2&t=1758&hl=slow+scsi+performance&s=9f0e65a3ff482 2032e4a63091694cc3f it never got fixed. Non-RAID IDE is unaffected supposedly due to a "bug" in the system where IDE devices ignore the OS commands to switch to write through caching. It's really ridiculous when a 700MB file takes almost 2 minutes to copy under XP and yet under BSD on the same system dual booted on the same array, it takes 11.2 seconds.
See http://forums.storagereview.net/index.php?act=ST&
Introducing the new Occam Fusion! Now with sqrt(-1) fewer blades!
1) NT4 (certainly from SP3) allows you to make partition alterations without a reboot. Even 2K requires a reboot for alterations to the boot partition, however.
2) 2K doesn't dispense with the drive letter concept, despite the implication in the article that this is the case. That you can mount partitions under folders doesn't change this.
3) You can specify the cluster size when formatting a drive under NT4.
Also, has anyone actually come across a data centre that is making use of multi-hundred-TB NTFS volumes?
And, will Longhorn finally do away with the whole drive letter concept?
"God, root, what is difference?" - Pitr, userfriendly
>find . -type f -iname "*.mp3
I think BeOS has plugins for different file types. IIRC it reads the tags in mp3s, so you could do queries on artist, genre, track #, album, etc. that might or might not be in the filename itself.
Yeah, but I bet you it'll be like Windows XP's theoretical ability to use FAT 32 - it can read it, but it won't actually let you format in FAT32 when doing an installation
So when is the last time you actually used WindowsXP? What you said is pretty silly.
You can install WindowsXP on FAT32, even create FAT32 partitions during setup etc, all the same options you get with NTFS during the install.
Are you trying to make up stuff or just never used it?
Geeshâ¦
yeah, but you still get a choice--i don't use mac os x's journaling because of the overhead--you don't hve to use winfs if the performance penalty is too high.
If done right, journalling has little to no performance impact. NTFS has had journalling since its beginning over 10 years ago, and even 486 based machines performed just fine with NTFS.
WindowsXP and Windows2003 Server of today still use NTFS and it is still fully journalled, and the performance numbers Windows2003 Server is getting shows that NTFS is not slowing the OS down a bit.
I'm sorry that the Mac Journalling has a noticeable performance hit for you.
From what I've read and experienced, the performance hit only involves writes. Reads are unaffected.
Writes are about 10% less efficient, which a pretty good tradeoff for that peace of mind.
--
the strongest word is still the word "free"
Then at the end there's a few paragraphs with no real info about the FS at all. What meta info will be stored? How will the files be laid out on disk? Is there going to be journaling? How about file system integrity and recovery?
The only thing _really_ learned was that there exists a 20MB beta executable that doesn't do anything. What the frell? It's two years before Longhorn is to be released. (As if Microsoft is going to get it right the first time anyway).
Mod the whole article down. Way down.
Worse than that, THG can't even get their facts straight. For example, when discussing fsutil.exe on page 4, the caption of the picture calls it a DOS app (it's not) and say it's from Sysinternals (perhaps they meant ntfsinfo, like the picture shows), yet the article text properly calls fsutil a "command line utility" (which it is) from Microsoft (which it is). While they do mention that it works on XP and not Windows 2000, they don't bother to mention that it's also available on Windows Server 2003, and that it's a system utility that's installed with the OS (c:\win[dows|nt]\system32\fsutil.exe). And just to add insult to injury, the "fsutil fsinfo" command they suggest you run is not quite correct. You need something more like "fsutil fsinfo ntfsinfo c:". "fsutil fsinfo" by itself just gives you another help screen, and not "scads of fascinating statistical information on the file system, volume and MFT."
All this article does is reinforce my dislike for Tom's Hardware Guide, and gives me ammunition I can use to convince others that THG is crap, too. If you want good hardware reviews, go somewhere good like AnandTech or Sharky Extreme. Hell, you could even go to Blue's News for the daily Hardware Reviews and still get better info. (I've not once seen Blue's link to THG from the Hardware Reviews ... I wonder why?)
The limit is 32GBs.
FAT32 Formatting in Setup is an option, but is known to frequently fail or have significant errors
No it doesn't.
FAT 32 Formatting once you are in the OS requires 3rd Party wares.
Again, only if creating partitions bigger than 32GBs.
Falsehoods on all counts, I have 2 machines running FAT32 partitions of up to about 20GB with absolutely no problems. It's not an issue.
Hmmm.
Uhh...no.
Slammer exploits a buffer overflow in the sql server "named instance" resolution mechanism. It has nothing to do with the storage/querying/indexing/etc of relational data.
The sysinternals driver is NOT a reverse engineering, it is basically a DOS mode interface to the windows system dll's. When you use the NTFSDOS disk you have to run it under the version of windows for which you will be accessing the NTFS partition, it then snarfs up the system dll's and puts them on the disk. Then when you load up the disk you run ntfsdos.exe which is basically a wrapper around that code. This could probably be done for linux but it would probably not be redistributable for fear of legal reprecussions from MS (how sysinternals gets away with it I do not know, maybe they have an agreement with MS, or maybe they are just too small of fish and too usefull for MS to swat them)
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Personally, I still have reservations about using a relational database to keep track of files. Unless they can keep the overhead to a minimum, I can't see it being as efficient as a file system should be.
Microsoft already has a file system that is integrated within a relational database. If you install Exchange 2000, you will see an M: volume on your local drive. You can flip thru this using Explorer as if it were an actual drive. In fact, it is a representation of a back-end relational database. The speed seems adequate to me so long as you have less than 8 GB in your mailboxes and public folders.
jwg
Plenty of filesystems are out there that do ACLs. FreeBSD's UFS2 does, I'm pretty sure SGI's XFS and IBM's JFS do also.
I think you miss the point of WinFS. It is actually a layer abover NTFS, so the actual files will still be stored the same way they are presently (and NTFS is pretty good about reliability). NTFS is already a lot like ReiserFS even without WinFS.
What WinFS adds is more powerful indexing, not a new storage system. Whenever you add or modify a file, WinFS adds the file's attributes to its indexes. The attributes stored are customizable, and vary depending on file type (MP3s have their ID3 info indexed, etc.). For example: somerwhere on my 120 GB disk I have a file named "code.txt" but it will take 10 minutes to find it by scanning the directory structure. Instead, I do a "SELECT Path FROM Files WHERE Filename='code.txt'" and WinFS comes up with the answer right away. If I have full text indexing, I could search for a specific phrase. Even more useful, you don't have to make playlists anymore. Just put all of your MP3s in the same directory, and when you want to hear all of your Bon Jovi, just perform the appropriate query. (Obviously you don't have to know SQL to make this useful.)
Some of this is already present in a more limited form in Unix. For example, I still can't figure out why Windows doesn't have something like the LOCATE database that is set up by default on my FreeBSD box. But WinFS will blow LOCATE away (update is real time, not daily, and it has much more than just the file name).
However, from what I've heard, so far it is a bit of a dog performance-wise. I hope it gets up to speed by ship time... At least it can be turned off if you can't take the perf hit.
Time flies like an arrow. Fruit flies like a banana.
You cannot format a volume larger than 32 gigabytes (GB) in size using the FAT32 file system during the Windows XP installation process. Windows XP can mount and support FAT32 volumes larger than 32 GB (subject to the other limits), but you cannot create a FAT32 volume larger than 32 GB by using the Format tool during Setup. If you need to format a volume that is larger than 32 GB, use the NTFS file system to format it. Another option is to start from a Microsoft Windows 98 or Microsoft Windows Millennium Edition (Me) Startup disk and use the Format tool included on the disk.
I've done it myself, with and without SP1 slip-streamed, and it should have let you do it as well. Are you certain the partition was only 30GBs?
Commercial licenses for ReiserFS are available; according to Hans Reiser, Namesys makes most of its money from NAS vendors who don't want to reveal that they use ReiserFS.
I use journaling and still get filesystem corruption that is not automatically fixed (like overlapped extent allocation) once in a while. On the other hand, HFS+ seems to already use a B-tree and file search is quite fast.
It should be noted that the regular fsck should still be performed, even on a journaling filesystem. The fsck after an unclean shutdown is obsoleted by the journal, but the regular fsck is supposed to catch subtle errors in the filesystem code and hardware problems which can cause the filesystem structure to rot slowly - problems which a journal can't and won't fix.
What is XP Pro and XP Pro Corp? There are no such versions of the OS.
For an overview of the major diffrences between Home and Pro
Apparently you just proved yourself wrong. As for XP Pro Corp, the Professional Corporate edition is basically the same as Professional minus product activation (you still need a serial number, just none of this registering with Microsoft crap).
I'm posting this from a WinXP machine. Let me assure you that while it can read FAT32, the OS as default will not FAT32 format any drive over (I belive) 2Gb.
w to /gettingstarted/guide/installnew.asp
/?
/N:sectors] /FS:filesystem Specifies the type of the file system (FAT, FAT32, or NTFS).
FAT32 Formatting in Setup is an option, but is known to frequently fail or have significant errors.
FAT 32 Formatting once you are in the OS requires 3rd Party wares.
SCO has performed an illegal operation and will be shut down.
I hate to rain on your parade, but the information you are providing you are either mixing with NT4 information or just making it up as you go.
Nothing I can say is convincing you, so let me reference you to the information you need.
http://www.microsoft.com/windowsxp/pro/using/ho
It clearly provides information on the only limitations of doing a FAT32 installation of WindowsXP, and that is the inherent limitation of FAT32 only being able to recognize a 32GB partition size.
As for '(I belive)2Gb', you are referring to the FAT16 installation of NT4. It doesn't apply to WindowsXP.
FAT 32 Formatting once you are in the OS requires 3rd Party wares.
Here let me quote from a command prompt for your edificationâ¦
Format
Formats a disk for use with Windows XP.
FORMAT volume [/FS:file-system] [/V:label] [/Q] [/A:size] [/C] [/X]
FORMAT volume [/V:label] [/Q] [/F:size]
FORMAT volume [/V:label] [/Q] [/T:tracks
FORMAT volume [/V:label] [/Q]
FORMAT volume [/Q]
volume Specifies the drive letter (followed by a colon), mount point, or volume name.
Notice the last line?
So where do you get your information, tooth-fairy, or just make it up as you go? Third party utilities are NOT needed to format FAT32 in WindowsXP, as you said âonce you are in the OSâ(TM) â" BTW, did you know that even during Setup â" YOU ARE âIN THE OSâ(TM)? It is running a stub version of NT already at that point; you are already in the NT OS.
FAT32 Formatting in Setup is an option, but is known to frequently fail or have significant errors.
Since when? Where do you document this at? Funny in all the time our tech team has put into working with WindowsNT, they can find no reference to this, either online or in their own experience. Tooth-fairy again or just making it up as you go?
There are NO known issues or documented issues of WindowsNT EVER having a problem running on a FAT partition, either FAT16 from NT4 days or FAT32 of current days. The NT core has an installable File System, like most modern OSes, and its interaction with the file system underneath is irrelevant to how it operates in regard to failure or problems.
Besides, this is really a moot point. If you are running WindowsXP and it is your only OS, then there is NO reason not to use NTFS. It is faster with larger files, faster with large volumes, supports compression, supports encryption and can have partitions up to 16 exabytes in size.
If you are running WindowsXP on FAT32, you are missing half the benefits of WindowsXP and the security of NTFS as well as its reliability of sitting on a journalled file system, etc, etc.
Geesh.
NTFS is based on attributes. The filename is an attribute of the file. The ACLs are an attribute of the file. The file data is also an attribute. Some attributes are special and are used by the file system itself, but otherwise a file is just a collection of attributes. So NTFS has always been able to act as a database in the way you mention -- the attributes in Win2K are simply stored as NTFS attributes.
The only thing missing has been indexing. NTFS is not designed to index everything the way a database does. And it doesn't have a query engine. This is what WinFS will add.
Time flies like an arrow. Fruit flies like a banana.
"you'd realize that Microsoft wouldn't go this far just to stop Linux from supporting NTFS. They'd just change a few important things about how NTFS works. They wouldn't throw it out unless there was a good reason to do so. And no, Linux having nearly decent support for NTFS doesn't qualify as a "good reason"."
Well according to their past history this is exactly what they would do. Messing with NTFS might break backward compatibility.
Why do you think MS has changed their stripes? Same management, same philosophy, same monoply. They are the same company as always and there is no reason to believe that they will act differently today.
War is necrophilia.
FAT32 Formatting in Setup is an option, but is known to frequently fail or have significant errors
No it doesn't.
Just a side-note, when you format a drive as FAT32 using XP/2k, Win98 won't recognise it and won't let you install to it ("Bad Media Block"), eg. when you want a dual-boot between 98 & 2k.
Also, if you have a look at the Partitions while using a Linux-based partitioning app (eg. Disk-druid, fdisk) you'll see a whole stack of < 1MB or < 4MB partitions everywhere.
Yurk. fdisk all the way.
I think you are confusing the terms. Or you just got the threads mixed up.
Journalling makes sure that the state of data, or meta-data is always defined. So even if you turn the computer off while in the middle of a write command the file-system will be in a correct state afterwards. (The write operation will be lost however.) It is basically a way of making HDD operations atomic. Either they succeed or they fail, the in between state is what puts your drive in a corrupted state.
When talking about 10% it is refering to space on the disk for the log file to make this possible.
Now storing the meta data in a database, which is essentially what WinFS and such are doing, is not as clear a benfit. Personally I can imagine that it would be a very practiacal FS for keeping movies and MP3's on. I don't really see the benefits of running the OS files on that FS though. A lot of unneccesary overhead. (I don't search for files in my OS partition very often.)
AFAIK, the NTFS installation of NT4 won't allow you to have a primary (system) volume > 4GB. This is because NT will install a FAT16 volume and later convert it. This may have been fixed in a service pack, but until you install the OS, how are you going to get a SP on there? Sure, you can
grow the partition later, but you're being a bit disingenuous in your specifying that this problem is confined to FAT16 installation on NT4, since an NTFS installation *uses* the FAT16 installation.
uhmmm.. XP's 2k mode is just for the widget set; makes it look like "classic" instead of luna.
No functionality lost. Performance and efficiency gained. All good.
ADM291 Tour of the Sysinternals Tools The Sysinternals.com web site is source of dozens of powerful freeware tools that reveal aspects about the internal behavior of the Microsoft® Windows® operating system as well as the activity of applications that run on it that are otherwise not available, making them valuable troubleshooting and administrative tools. Many are used on a daily basis with in Microsoft's product support and development groups. Take a tour, guided by their author, of some of the most popular tools. You'll learn undocumented tips, see examples of how to get the most out of them, and learn about how they work. Speaker(s): Mark Russinovich (Winternals Software)
"I love deadlines. I love the wooshing sound they make as they fly past" Douglas N Adams
This is about denying Linux and BSD compatibility to their file system. NTFS is being reverse engineered as we speak, write access is being worked on, so obviously Microsoft needs something new to get rid of them pesky Linux people.
All the new and great features could be done with a (simple?) layer between NTFS and the application. There is no reason why Microsoft needs to invent a new file system here.
The Windows Search function with indexing service turned on finds files practically instantaneously -- when it actually uses the index. The trouble is, even with indexing service turned on, it rarely uses the index unless you force it to. You have to put an "!" in front of the text you are searching for. Not that it tells you this anywhere, of course. It puzzled me for about a year before I finally stumbled on this:
http://xpsearch.info/xpsearch.htm
Someone actually made a site specifically about this silliness.
This applies to Windows XP with Indexing Service turned on, which searching for text inside files. I would tend to think it applies to 2000 also.
Note that you need to be running NTFS for indexing service to take advantage of file tracking to determine when files are created/modified. (Does indexing service even work in FAT32? Does it scan the whole drive periodically?)
At any rate, I have to say, once you know about the "!" and also occasionally take advantage of the query language, indexing service works so well that I'm kind of wondering what WinFS could do for me that IS doesn't already do.
Postgres as an rdbms backend to a filesystem front end.
3
Designed mainly for version control. Could easily be modified for other purposes though.
http://www.linuxjournal.com/article.php?sid=138
Government of the people, by corporate executives, for corporate profits.
AFAIK, the NTFS installation of NT4 won't allow you to have a primary (system) volume > 4GB. This is because NT will install a FAT16 volume and later convert it [is-it-true.org]. This may have been fixed in a service pack, but until you install the OS, how are you going to get a SP on there? Sure, you can
c id=kb% 3Ben-us%3B119497
grow the partition later, but you're being a bit disingenuous in your specifying that this problem is confined to FAT16 installation on NT4, since an NTFS installation *uses* the FAT16 installation.
Nice information, but it has nothing to do what I was talking about.
If you read the post, you will see that I was responding to a post that said WindowsXP had a 2gb partition limit for FAT32 installations. Which is NOT true.
I never said anything about the primary boot partition of NTFS on NT4 or its size.
In reference to what you said, you are partially correct, SETUP of NT4 and previous versions will not allow for a boot partition larger than 4GB; however, you can pre-partition a Hard Drive with NTFS using another WINNT computer to whatever size you would like and then install WINNT4.0 on the blank larger NTFS boot partition. (This is a well known workaround in the industry.)
As an additional note the original Pre NT4.0 setup did not fully load the NT Kernel and related drivers to support the drives or the NTFS file system, this is why the installation created a FAT partition for the file copy portion of the install. NT4.0 did fully load the NT Kernel, but the original installation mechanism for partitioning was left in from the NT 3.x days.
Windows2k and WindowsXP do not have these restrictions and fully load the NT Kernel during the initial file copy portion of the Setup.
People wanting more information try:
http://support.microsoft.com/default.aspx?s
Take Care,
The Net Avenger
The stupid thing is, despite the fact driver-letters are now only a backwards-compatibility hack in the Windows kernel VFS, you can't use UNC paths at the cmd.exe prompt. They even went to the trouble of coding a specific error message to tell you not to! God forbid Windows should ship with a useful CLI! (Actually, I like to code up a simple Read-Eval-Print-Loop in JScript and run it under the WSH if I'm on Windoze, but that's hardly an out-of-box solution)
P:\>cd \\myserver\myshare
'\\myserver\myshare' is an invalid current directory path. UNC paths are not
supported.
WHY THE HELL AREN'T THEY SUPPORTED! MS ARE TEH STOOPID!
Take a look at www.namesys.com/v4/reiser4_the_atomic_fs.html
and at www.namesys.com/v4/v4.html.
We will be adding support for semi-structured data querying in the next major release, assuming that we find funding for it. The semantics for it are described at www.namesys.com/whitepaper.html, which also explains why I don't think the relational model is effective for semi-structured data stores such as a general purpose filesystem is normally used for.
Best,
Hans
Avoiding deadlock, dealing with transaction timeouts, controlling permissions on who can keep a transaction open for how long, these are all very serious issues that have us first making transactions available to plugins and trusted processes only in the first release. However, there is a LOT you can do with just plugins and trusted processes to make user data more secure.
I noticed right clicking in Win Explorer in Win2K allowed you to apply attributes to files. Not sure if they were limited to MS Office files.(Sorry, I can't check this anymore ;-)
.AppleDouble directories for an approach to foreign datastreams...they can get ugly pretty quick.
I believe that NTFS supports multiple streams of data per filename much like HFS. However, Windows doesn't make the pervasive use of the capability that Apple did in OS 9. I think one reason is that pretty much anything can go in these multiple streams meaning that particular files are tied even more closely to particular applications. Another reason may be that multiple filestreams complicate transfering files from machine to machine.
You're correct about RPM. It is using a more conventional approach by chucking it's data in a database file. Read about Netatalk's
I don't think it is possible to configure ReiserFS wrongly and to lose data as the above suggests. What is very possible is to use it with the wrong kernel and obsolete utilities. If you use ReiserFS with RedHat, get an OFFICIAL kernel from Marcelo, not a redhat kernel, and get the latest utilities off of our website.
I would take a guess that the users complaining about losing data are redhat kernel users and the ones that are happy are SuSE or official kernel users. When you compare stability of ReiserFS and ext3, try to compare them using the same version of the official kernel. We have generally been more stable than ext3 in the official kernels at any given moment in time, in large part because we were about 6 months ahead of them in the development cycle. Remember that RedHat does not tend to keep up with ReiserFS updates in its nonofficial kernels.
Recent RedHat kernels are probably much more stable than the old ones because they have bugfixes from more recent official kernels, but remember that these are the guys who in one of their releases compiled reiserfs with debugging turned on to make it slower than ext3, so I would go with an official kernel if it was me.
V3 of reiserfs is very stable currently. We go for months without bug reports and we have a lot of users in Europe. This is in large part because we put all our new features into V4. V4 will be unstable for quite some time.
There are database products available out in the world that take a snapshot of a filesystem based on a time interval. All changes after this time do not get added to the backup, even if it's the last file to be backed up, and it's brand new.
This is standard fare in the database universe, where a change to a record after the backup has started can break tables already stored to tape, therefore destroying referential integrity. Many of the problems we have with backups and system restores today could be alleviated if our filesystems had this functionality built into them naturally. In many systems it goes by the acronym MVCC (multi-version concurrency control).
It would allow me to kick off a backup, and make changes to system settings, and yet the backup system wouldn't pick up any new settings, the filesystem would queue the changes until the backup "transaction" had ended, and then commit the new changes permanently.
Windows has a product available for it now that does this, I'm not sure of the name for it though. I have to tell you, the ability of linux to do this at the VFS layer would be killer. Of ANY OS really. And it's come to the point where software is so complex, and so interdependant on the OS (Windows particularly) that it SHOULD be a standard feature.