CNet on WinFS

Weston writes "CNet has posted an article about WinFS, or more specifically, what Bob Muglia (a VP at Microsoft) said about it in a recent interview. According to Muglia, the new filesystem will not replace NTFS, but will incorporate feratures of NTFS, SQL, and XML all into a filesystem which, accoring to Microsoft, will open up a whole new world of information availability. He goes on to describe such a filesystem as the 'holy grail' that is sought by developers. WinFS is slated for release in 2005/06 as part of the Longhorn OS."

Comments

My Docuement into My Database

[ericspinder]

Instead of having "My Documents" Windows users will have "My XML-based Database". A nice GUI for making queries and they'll remove the "My Computer" icon (or at least hide it more than XP.

I suspect little other changes, perhaps they'll also make it the "next registry", and that is why it will take so long or may it they are waiting for 5ghz chip so it won't seem so slow.

Re:My Docuement into My Database

[TedCheshireAcad]

Well, as slim on technical details as this article is, here's what I've got:

Another key building block is the querying capabilities of Microsoft's SQL Server relational database, according to Microsoft. WinFS also will incorporate the data labeling capabilities of Extensible Markup Language (XML), Muglia said.

This seems like way too much overhead for a problem that can be solved with some clever indexing.

"Today, applications encapsulate data. In the future, applications will be able to read and write data created with multiple applications," Muglia said. "Information opens up dramatically."

Applications sharing data? Can't that happen when you don't use a proprietary file format for say, an Office suite?

They're blowing some serious smoke up the typical PHB ass, but then again, when aren't they?

Re:My Docuement into My Database

[tuffy]

Applications sharing data? Can't that happen when you don't use a proprietary file format for say, an Office suite?

Perhaps Microsoft should invent some sort of Standard Code for Information Interchange. It'll be a big hit, I'm sure.

Been saying it for years

[Perianwyr Stormcrow]

Filesystems are just inefficient, shitty databases.

Re:Been saying it for years

[Frymaster]

Filesystems are just inefficient, shitty databases.

if by efficiency, you mean "speed of read and write" then i don't think this is going to be an improvement. the article makes it sound (although it's short on details) like winfs is just a front-end for ntfs and sqlserver - another layer your read-and-or-write has to go through before it gets to the disk.

but, hey, it's got xml for buzzword compliance!

Re:Been saying it for years

[uberdave]

Instead of just saying it, you should have done something about it. Imagine the riches you could be rolling around in if you had sold a revolutionary new filesystem to BillyBoy. Or imagine your name spoken with the same respect and awe as Torvalds, etc, if you open sourced it.

Re:Been saying it for years

[orthogonal]

Filesystems are just inefficient, shitty databases.

And sometimes all I want is a shitty database.

What worries me is that Microsoft already overrides my wishes; if they think they've created the "holy grail", they'll be even more likely yo impose it on me even if I know it's not what I want.

Case in point: Windows 2000 and above has no problem reading FAT32 partitions greater than 32GB in size. But it refuses to create FAT32 partitions > 32GB in size. Why? Because at that size, Microsoft knows better, Microsoft knows you should be uses NTFS and get the benefits of meta-data and journaling.

Except, I don't want the overhead for my MP3 collection. The meta-data's already present in the ID3 tags, and I don't need journaling -- once the ID3 tags are written, they're essential read-only. I want low overhead storage for very large (several MB) files.

And I want something that is a mirror of my portable plaayer, which can only read FAT32, can only read the first partition, and is 60GB. Since my portable only reads FAT32 (but doesn't format), and since Microsoft, in its wisdom knew better than to allow me to format it as FAT32, instead I got to watch it run the drive for over an hour before telling me the partition was too big. Talk about a linux killer app: I nearly had to switch to linux if I wanted to be able to use my portable.

Fortunately for me that the open-source world exists: somebody had actually compiled the linux mkfs for cygwin, and I formatted the portable with it. I can't use Windows' chkdsk on it, of course, and I haven't yet looked at compiling fsck under cygwin to further work around Microsoft's collosal arrogance.

Given my experience, if Microsoft thinks they have the "holy grail" of filesystems, it, and Microsoft's arrogance, will once again be rammed down the throats of every Microsoft user. But by that time, I'm sure I'll have fully transitioned to linux.

Re:Been saying it for years

[gpinzone]

So use FDISK. It's still free if you own Windows.

Re:Been saying it for years

[grub]

What worries me is that Microsoft already overrides my wishes; if they think they've created the "holy grail", they'll be even more likely yo impose it on me even if I know it's not what I want

They can only impose their will on you if you let them. You can just say "enough" and move to a Mac, Linux, *BSD, et al.

Re:Been saying it for years

[SpaceCadetTrav]

The overhead for NTFS vs FAT on mp3 files would be so minimal that you would never notice. Get over it.

Re:Been saying it for years

[penguin7of9]

Filesystems are just inefficient, shitty databases.

The fact that file systems are databases has been recognized since, oh, databases were invented. One of the first things IBM tried after inventing the relational database was to replace the file system with it. You can tell how far that went.

The choice to make the UNIX file system the kind of database that it is was deliberate. UNIX file systems are a highly efficient and robust database, with proven metadata, security, and data consistency models. They do almost exactly what people want databases to do with their unstructured data.

For anything else, they use other databases. By a stroke of genius (or maybe just historical inevitability), those more specialized databases can be stored and accessed inside the file system database.

A database file system is quite accurately described as "The Holy Grail": it's an ancient mythological object of no practical value, something that only insane people would pursue.

Re:Been saying it for years

[prisoner-of-enigma]

"Case in point: Windows 2000 and above has no problem reading FAT32 partitions greater than 32GB in size. But it refuses to create FAT32 partitions > 32GB in size. Why? Because at that size, Microsoft knows better, Microsoft knows you should be uses NTFS and get the benefits of meta-data and journaling."

There may be a far-less-nefarious reason why it won't let you create a FAT32 partition of size >32GB, namely that the FAT table would be ridiculously large and inefficient.

Now, you could argue that the Master File Table for NTFS is also large and inefficient, but at least you have some control there over cluster sizes. Don't like the default 512 byte size? It goes up to (I think) 16K in size. Sure, you end up getting some disk storage inefficiencies but you can get around this by either (a) picking a cluster size more in tune with your actual intended usage or (b) using file system compression which attempts to clean up a lot of hanging cluster wasted space in addition to its compression duties. The fact that you get security, journaling, and far better error recovery than FAT32 is just a bonus.

Re:Been saying it for years

[Worminater]

uh...

if you actually READ his poast(i know, something lacking in many comments...) you would notice that his portable MP3 player would only sync with a fat42 file system.

he wants all his mp3's on a 60gb hd

windows refused to let him install fat32 on that drive, so he was officially screwed in what he wanted to do.

Re:Been saying it for years

[eric2hill]

One of the first things IBM tried after inventing the relational database was to replace the file system with it. You can tell how far that went.

What do you mean? To this day IBM uses a database AS the filesystem in the AS/400. Ever heard of DB/2?

Re:Been saying it for years

[__aafutm5472]

A database file system is quite accurately described as "The Holy Grail": it's an ancient mythological object of no practical value, something that only insane people would pursue.

I agree. My pastor at church described a situation where some Bhuddists found what they believed to be a peice of Bhudda's finger. They put it on display, and millions of Bhuddists from around the globe made the trek to see it.

But why? Did it solve their problems? Did it bring them closer to Bhudda? Not saying it isn't something that might be cool to see, and it may have even been inspiring to see all those people flocking to it, but the finger itself didn't provide any spiritual oneness.

I may be wrong, mind -- what do I know, really, of Bhuddism? Or filesystems, for that matter. What I do know is if Microsoft says it's the "Holy Grail", then they're arrogance has grown beyond belief...

Re:Been saying it for years

[dasmegabyte]

Why does it matter? The processor overhead caused by NTFS over Win32 is negligible when reading mp3s off of a disk. The maximum speed you can read data off the disk is 100 million bytes per second, while your average processor performs around 2000 to 3000 million operations per second. You're talking about a difference of a few (certainly less than 100) operations per second of overhead...something you will never see, except in extended throughput. It is certainly nothing compared to the overhead of playing an mp3 or even copying a file.

In fact, the only reason I can thing of to use Fat32 is if you need to support Windows 98, or some other system that can't read Fat32 (eg your mp3 player). Metadata knowledgable storage systems will be VERY handy in multimedia systems and what you mention about ID3 tags is exactly why it'll be useful...instead of having to read a bit out of each file (with all the IO that entains) before you can search on these fields, you can read out of a single area. We use ID3 tags because storage is currently ignorant of content. We use databases because each ID3 is ignorant of all the others, and they are therefore ungrouped. This new file system may have more "overhead," but it'll get recouped fast and you will never, ever notice it. Your hard drive isn't a racecar, man. Err on the side of convenienve.

Oh, and you can easily format >32 GB using the command line format. I just did it. I am not against MS preventing users from doing something the easy way if they have another way of doing it...especially when it promotes

Re:Been saying it for years

[NanoGator]

"Microsoft knows you should be uses NTFS and get the benefits of meta-data and journaling."

Do you have any idea what it costs to support every little thing any given company makes over time indefinitely?

Just a thought. You don't need to throw the word arrogance out there.

Re:Been saying it for years

[Jucius Maximus]

"Case in point: Windows 2000 and above has no problem reading FAT32 partitions greater than 32GB in size. But it refuses to create FAT32 partitions > 32GB in size. Why? Because at that size, Microsoft knows better, Microsoft knows you should be uses NTFS and get the benefits of meta-data and journaling."

"So use FDISK. It's still free if you own Windows."

No, don't use FDISK. There are too many bugs, particularly when you're handling logical drives in extended partitions of and the last partition is of a type that it does not recognise (i.e. linux partitions.)

FDISK has 'accidentally' destroyed partitions of my own data due to this bug. Instead, I suggest you check out the excellent and GPL partition manager called ranish partition manager which does a much better job than microsoft's FDISK.

But FAT32 is becoming less and less useful compared to NTFS. I don't even need it anymore on my win32/linux machine because Mandrake 9.1 reads the NTFS partitions right out of the box. No configuration was needed.

IMO the last remaining legitimate use of fat32 is on mobile drives that you have to use in a variety of machines. My iBook won't read the NTFS and the Win2k won't read the OS X file system. Gotta use FAT32.

Re:Been saying it for years

[FreshFunk510]

I'm well aware of this theory, but in the context of :

1) Increasing size and need for search on home computers.

2) Explosion of media and information on consumer machines.

3) Increasing need for data transparency across machines.

I'd say it's a task worth undertaking.

Does it mean that MSFT is trying to make it so that you could import your Sql SERVER or Oracle database onto a WinFS system? I highly doubt it. And that's the distinction that needs to be made.

It doesn't seem like they're converting the filesystem into an enterprise level database. They're just enabling the core features of a db (search) for a filesystem that is long overdue.

And, yes, this has been tried in the past but disk drive technology has come quite a way since then. I'm sure Google searches weren't possible back then either.

Re:Been saying it for years

[Jucius Maximus]

"There may be a far-less-nefarious reason why it won't let you create a FAT32 partition of size >32GB, namely that the FAT table would be ridiculously large and inefficient. Now, you could argue that the Master File Table for NTFS is also large and inefficient, but at least you have some control there over cluster sizes."

This is a limitation of Microsoft's format.exe utility. Maxtor drives tend to ship with their own utility and you can specify a nice 4K cluster size for your monster size drive.

Re:Been saying it for years

[Deflagro]

That's what i was thinking also. That FAT can only format up to 32GB. It's not the OS, it's the FS. NTFS can do up to like a petabyte or 7 PB or something. I'm not all for Winderz but that argument didn't seem logical.

Re:Been saying it for years

[MattRog]

The relational model is a general model of data. It can store anything.

And by the way -- data either has structure or it doesn't; the only useful unstructured data you'll find in a filesystem is the contents of /dev/rand.

Re:Been saying it for years

[orthogonal]

There may be a far-less-nefarious reason why it won't let you create a FAT32 partition of size >32GB, namely that the FAT table would be ridiculously large and inefficient.

It may be inefficient, though I don't think it need be. As I point out, I'm dealing almost exclusively with large files (and despite another reply, I think an average of 5MB per file is large, at least compared to say, 512 byte .txt notes), so the cluster size can be correspondingly large. Larger clusters, of course, mean a smaller FAT table.

Be that as it may, my MP3 player only reads FAT32 partitions. However efficient NTFS might be compared to FAT32, if I can't use NTFS, it's not a good solution for me.

The fact that you get security, journaling, and far better error recovery than FAT32 is just a bonus.

Again, it's no bonus if I can't use NTFS. There's no technical reason that FAT32 partitions larger than 32GB can't be written -- mkfs will write one -- or read -- even Windows 2000 will read one. It's a purely arbitrary limitation on the part of Microsoft, because, they, like you, apparently never thought there'd be a legitimate need for a FAT32 partition larger than 32GB.

And that's my beef with Microsoft; and again, I was using this arbitrary limitation as an example of what's wrong with Microsoft's thinking: a program shouldn't arbitrary limit the user based on assumptions made by the programmer, especially when it actually involves extra work to add the limitation.

It's like Clippy interrupting you, and saying "It looks like you're writing a letter! I'm going to add your return address line block of whether you want it or not! And I won't let you delete it!"

Come to think of it, that's pretty much what Word's auto-correct does. Try starting a sentence "e. e. cummings said...." or "Intentionally misspelling 'the' as 'teh' frequently signals sarcasm in /. posts" in Microsoft Word, and notice how Microsoft assumes capitalization and spelling that is not what you wanted.

Re:Been saying it for years

[MagicBox]

well saying 'enough'and moving on is easier said than done. Just because I have a problem is that the reason to drop everything I know (basically my career) and start all over again at 30 years old? I do not see that as an easy thing to do. But then if I moved to *nix will it solve my problems? Probbably some, but I am sure I'll still have other problems. Bottom line? You have to come up with a solution. As a techie you should know that you'll always have computer problems. If windows cannot do something, then use Linux (it's free) and whatever Linux cannot do, then use Windows to accomplish that. Instead of extinction I see co-existence. Will the fate of both OSes eventually follow that of the Cro-magnon man(thought to have interbred with Neanderthals)? Time will tell.

Re:Been saying it for years

[TheNetAvenger]

Ok, people were up in arms that NTFS would have SO much overhead that it would never be viable in desktop PCs(1992 Buzz). And here we are again...

However, even on the 486-33mhz 16mb Systems that NT 3.1 and 3.51 were released on showed that NTFS's performance was at the same level and sometimes much faster than less featured FS technologies like FAT, etc.

Ironically, Journaling is still seen as a major overhead on OSes like OSX - which baffles me, considering MS was able to offer great journaled performance on a 486 platform, and on modern systems is transparent and offset by the advanced MFT and data structuring in NTFS.

and I don't need journaling

You don't? Do you even know what it is? Journaling allows the OS FS to track and protect from read/write file corruption. Especially in cases of power loss or hardware failure.

This is why CHKDSK screens on NT running NTFS are very rare - the journaled NTFS takes care of these problems without letting the FS or the information on the FS get corrupted by an interrupted write for example.

File Systems are a database variant, whether people like it or not. Why not add the advanced indexing and heuristics of a database technology onto the basic NTFS structure (especially since NTFS was designed to be extended in this direction).

And no, a simple indexing server is not going to cut it, there are simple indexing systems in Win2k and XP already, they prove inefficient and do not expose non-FS storage data mechanisms to applications.

Also your Metadata example of the MP3s is already a moot point. NTFS using Metadata ALREADY, just because it is not that obvious to users, does not mean that NTFS is not ALREADY doing this. This is part of the design of NTFS, the ability to store more than name, date, and location of file structures. It can store virtually any amount of additional data about files on the volume, and already does with many files types. People just don't realize that this is already happening, and because NTFS was designed to do this, there is already NO performance degradation.

Put fast information retrieval technology that exists in Database servers behind this, and not only will you add functionality for users, but it could in theory also increase the file access performance of the NTFS file system itself.

Database server algorithms are very fast and efficient and offer relational constructs, something a modern OS just might be able to use - especially when you stop seeing files and documents on a volume as closed entities and instead realize that they are data constructs that often relate to other data on the system. (i.e. A very easy basic example would be Mail Folder Files)

Re:Been saying it for years

[Molt]

Reading what he says he wanted this partition for, it sounds like he's right. All of your points for NTFS being superior to FAT32 are correct in general, but in this particular case are not convincing.

The partition is meant to be a mirror of his MP3 player, and I'd hope he has his collection burnt to CD/DVD somewhere. Journaling isn't that major here, data loss isn't too great a risk compared to having to use a FS which isn't an exact mirror of his player's. If there's hardware failure then hit the shiny pile of CDs and copy them back, no biggie.

The metadata is already in the ID tags of his MP3s, and in a way that his player can understand. Why would he want NTFS adding extra metadata which'd get lost when he moved the data across? Performance is secondary to practicality here, and very much so, he's after something which is like his player and not an optimised MP3 retreival system.

You're right about database algorithms being very fast and offering many nice features, but in this case the features are not only not useful but they're a positive hinderance. His player will only sync with a FAT32 partition, no matter how many nifty bells and whistles NTFS offers it's still a FAT32 partition his player wants. Windows will not let him create this, not merely advise against it but not allow it, and as such I view this as a flaw.

Note that I'm not generally a Microsoft-basher in any sense.. I actually quite like Windows 2000/XP despitie this being a less-than-popular view here, but in this case there is a definite and unwarranted stupid restriction.

Re:Been saying it for years

[grub]

Just because I have a problem is that the reason to drop everything I know (basically my career) and start all over again at 30 years old?

I'm almost 38 and have no problem picking up new things :P

of course your concerns are valid, but learning on your own at home will have you looking for ways to apply your new knowledge at work.

Re:Been saying it for years

[toddestan]

FAT32 can do upto 2 Terabytes, and I remember Microsoft making claims to that back when they were pushing Windows 98.

It is true that after 32GB you have to use 32k clusters though, which is not that efficent (and why FAT16 just sucked for 1-2GB partitions). But for a collection of MP3's the loss of space would be minimal.

So yeah, the nerfing of the format utility is just another example of Microsoft forcing you to do things their way, cause that's the way they want it to be.

Re:Been saying it for years

[Talez]

The metadata is already in the ID tags of his MP3s, and in a way that his player can understand. Why would he want NTFS adding extra metadata which'd get lost when he moved the data across?

The argument works for MP3 because MP3 meta-data (ID3) is ubiquitous and I admit you make a very good case. However, I have to agree with the parent. Filesystem level metadata is a Good Thing. Format dependant metadata means you have to write a new damn parser for the metadata every time a new file format comes out.

IMHO, the way to solve the transfer problem would be to wrap the file in a metadata "wrapper" before its sent. Much like objects are serialized in Java or PHP.

Re:Been saying it for years

[spike hay]

Case in point: Windows 2000 and above has no problem reading FAT32 partitions greater than 32GB in size. But it refuses to create FAT32 partitions > 32GB in size. Why? Because at that size, Microsoft knows better, Microsoft knows you should be uses NTFS and get the benefits of meta-data and journaling.

Except, I don't want the overhead for my MP3 collection. The meta-data's already present in the ID3 tags, and I don't need journaling -- once the ID3 tags are written, they're essential read-only. I want low overhead storage for very large (several MB) files.

In this case, Microsoft does know better. NTFS has less overhead than FAT32. Sure, it has more meta-data, such as the file level security (not that that would EVER be noticable). But NTFS has smaller cluster sizes than FAT32, so at more than makes up for the meta data. Just use NTFS.

Re:Been saying it for years

[penguin7of9]

What do you mean? To this day IBM uses a database AS the filesystem in the AS/400.

Congratulations: you could inded tell how far that went, namely as far as a few niche operating systems.

Ever heard of DB/2?

Not only heard of, I have developed software for it. Incidentally, DB/2 (well, the UNIX version at least, the mainframe version is a completely different beast) has facilities in it to store large chunks of data in the file system--because it's much more efficient.

Re:Been saying it for years

[TheNetAvenger]

No, it's actually because Windows 2000 will develop registry corruption on large FAT32 partitions

This isn't a specific flaw of Win2k, or the registry. FAT32 is inherently not a data safe FS, the registry corruption is likely just a result of power loss or system failures when the OS is writing to the registry.

NTFS is journaled and protects against this type of write failures to ensure the registry will not get corrupted. FAT32 doesn't.

If you use ANY OS with a FAT/FAT32 file system, you will eventually find write failures from accidental power offs to basic system failures.

The registry hive is just a little more sensitive in the Win2k/NT line, as it is the core of the OS.

I have also seen registry and lots of other file corruption in non-journaled OSes. Cruicial system setting data corruption is just a prevalent in other OSes as well, it is just not centralized. Hunting down a corrupted config file in *nix for hours because of power failure on a desktop system is not a fun task either.

Luckily in Win2k you can use the recovery tools to restore the previous registry versions and not do the hunt and find to get the OS fixed.

However, as recommended with ANY OS, you should be using FS with certain safe guards like journalling. Period.

Re:Been saying it for years

[EddWo]

Then make sure you choose the right cup. Or you might find yourself getting very old very fast.

Re:Been saying it for years

[penguin7of9]

I'm well aware of this theory, but in the context of :

1) Increasing size and need for search on home computers.

2) Explosion of media and information on consumer machines.

3) Increasing need for data transparency across machines.

I'd say it's a task worth undertaking.

Putting any kind of standard database into the file system is not going to help with any of those. Search is already handled quite well anyway. Media (images, video, audio, etc.) are not well addressed by the kind of database Microsoft is proposing to put in. And how exactly is a database supposed to help with "data transparency across machines? That's something file systems know how to do and know how to do well, but the equivalent database functionality requires high-end, costly, and complex databases--in different words, the problem gets worse.

And, yes, this has been tried in the past but disk drive technology has come quite a way since then. I'm sure Google searches weren't possible back then either.

Funny thing is: Google is using a file system for their data storage, and they are definitely not using the kinds of databases that Microsoft wants to put in. If Google did use a design like Microsoft, they would be out of business.

It doesn't seem like they're converting the filesystem into an enterprise level database. They're just enabling the core features of a db (search) for a filesystem that is long overdue.

That's already there, on Linux, today. You don't need any more enabling.

Re:Been saying it for years

[FreshFunk510]

Putting any kind of standard database into the file system is not going to help with any of those. Search is already handled quite well anyway.

That's the thing. I think you have to cast aside the thought of combining a file system and traditional database. As I said in my original post:

Does it mean that MSFT is trying to make it so that you could import your Sql SERVER or Oracle database onto a WinFS system? I highly doubt it. And that's the distinction that needs to be made.

It doesn't seem like they're converting the filesystem into an enterprise level database. They're just enabling the core features of a db (search) for a filesystem that is long overdue.

So, without repeating myself, there it is.

Search, as it is now, is NOT handled well. Have you ever used Win 95/98/2000/XP? There's no useful search feature. If you want to find a document across an 80Gb hardware be ready to wait about 10 mins (more or less). When I say "good search" I mean a utility where I could go to and simply type "Jay Z" and instantly have the 10 most relevant files given to me.

Media (images, video, audio, etc.) are not well addressed by the kind of database Microsoft is proposing to put in.

Actually I don't think you can say that it can't (and, conversely, I can't say that it will). Why? Because they havent' given enough details on how this will actually work and without knowing that one cannot say. However, I think Microsoft is smart enough to notice that the explosion of media storage in the last half decade has really only been used to serve media (gifs, mp3s, and now mpgs). Storage is increasingly becoming media-centric.

My guess is that if they were going to do a useful "db" that serves media, they will have to use/create some standardized features for maintaing metadata regarding media (i.e. people in gifs, chapters in mpgs (a la DVD), and better metadata on text). All of this is quite possible and has not been done well.

And how exactly is a database supposed to help with "data transparency across machines? That's something file systems know how to do and know how to do well, but the equivalent database functionality requires high-end, costly, and complex databases--in different words, the problem gets worse.

Well, for one, I think it's a mistake to take current databases (SQL Server, Oracle, etc) and try to wrap that idea around a database. I dont' think that's what they're doing and that's what you're basing your arguments on.

The database, itself, I dont' think will help with data transparency. I said that in regards to the fact that much of hte metadata wil be maintained in XML (along with their comments of really globalized their use of web services).

Funny thing is: Google is using a file system for their data storage, and they are definitely not using the kinds of databases that Microsoft wants to put in. If Google did use a design like Microsoft, they would be out of business.

Again, you missed my point. My point in the original post was a response to the people who said "it's been tried before and it doesn't work". I was saying, "Look. Yes, a similar idea has been tried before but this was before the advent of really good disk drive technology. It's like saying something liek Google was impossible 10 years ago when, really, it was only due to the technology behind it." I say this because we learn, at school, about the endeavors made to combine filesystems and databases. But the context has always been around enterprise systems and enterprise use instead of individual consumer use. There are also other subtle differences that change the problem around (i.e. the primary type of data you're serving is media and not nice small chunks of text).

Anyway, another mistake you made was assuming that I waas saying that they were putting Google technology into Longhorn. All wrong. I was saying that they want to provide the same FUNCTIONALITY not TECHNOLOGY.

Re:Been saying it for years

[Minna Kirai]

What is a "Bhuddist"? I'll assume you meant to say "Buddhist".

My pastor at church described a situation where some Bhuddists found what they believed to be a peice of Bhudda's finger.

That makes little sense (not that I should expect any religion to make sense for long). In the Buddhist faith, "Buddha" is not an individual, but a category of person. It is a word used similarly to "saint". Of course, it's quite possible that the worshippers he described were not doctrinally correct, and had instead fallen into theism and idolatry. That happens with a lot of major churches- the people demand it, so the clergy provide.

Such behavior is quite similar to what Catholics have done for millenia (right up to the present). Does kissing the skull of a saint bring you any closer to Jesus?

I wonder why the pastor choose to use an alien, Asian religion in his example, when plenty of well-known Christian branches do exactly the same thing.

Developers, developers, developers!

[Scoria]

He goes on to describe such a filesystem as the 'holy grail' that is sought by developers.

These "developers" compose fluent VBS, I presume? ;-)

Re:Developers, developers, developers!

[inteller]

yes, and more quickly than python and perl developers.

my language can kick your languages ass.....whatever.

Re:Developers, developers, developers!

[RevDobbs]

Hmm, the holy grail that I seek will not loose any data on a suprise power-down.

Nor will it overwrite important data just because I told it to.

If it was really kick-ass, my file system of choice would also debug my code for me and produce valid HTML & CSS documents. And hide pr0n from my girlfriend.

Re:Developers, developers, developers!

[NineNine]

Hmm, the holy grail that I seek will not loose any data on a suprise power-down.



NTFS won't "lose" data when the power is cut off. I don't know if it'll "loose" data, though.

Nor will it overwrite important data just because I told it to.



So then you want a FS that *doesn't* do what you tell it to do? Somehow, that sounds like a really, really bad idea.

Re:Developers, developers, developers!

[RevDobbs]

The loose data is helpful for fuzzy math.

VMS has a file system that automatically does file versioning: saving a file actually creates a new copy.

Re:Developers, developers, developers!

[Steve Franklin]

I just want the morons to stop changing things every time they get even close to actually making the damned bloatware work properly. Do you realise there's a "fix" at their update site for all the things XP SP1 broke? Only it won't download because something else is broken. These folks are all idiots. Certifiable, institutionalizable, garden variety idiots. They need to put the damned OS on the shelf and start selling software that isn't tied into it just because that's the only way people will buy it.

Every version they've cranked out has been the "ultimate." So why is it they keep having to "improve" it? Next they'll want to go back to decimal computing just to confuse the developers even more.

Re:Developers, developers, developers!

[sketerpot]

That might be a good idea if you have a disk much bigger than you need. The filesystem could, say, write a new copy, and if it ran out of copy space it would overwrite the oldest copies. It's not something you could rely on, but it could be pretty convenient to undo changes you never anticipated having to undo.

Re:Developers, developers, developers!

[BRTB]

This is what Netware has been doing for a long time now. VERY useful... it's one of the only things I can think of that's missing from Linux as a server. The good old SALVAGE tool (and underlying filesystem support, I assume) has fixed things more times than I can remember.

Re:Developers, developers, developers!

[pottymouth]

I've been working in my parents basement too long! My ass is no longer structured!!

Of course working from home on my own schedule and making enough money to buy M$ scripters (not really a programmer if you work for Mr. Bill) by the boatload ain't so bad! Maybe you should try learning a little about what you try to do for a living and we'll all be better off. Then again, after you've spent years of writing crap I get called in to write a real program (for a really nice paycheck) so please, go ahead and keep VBing away!!!!

Re:Developers, developers, developers!

[bfischer]

VBS has a valid place in the windows world. It is very useful for admin tasks through the WMI.

Re:Developers, developers, developers!

[TomV]

And what's more NTFS will also hide his pr0n from his girlfriend, as requested ;-) Although wXP does go way too far in trying not to worry the user's pretty little head with that scary Security stuff.

The truly careful would PGPdisk it too.

Re:Developers, developers, developers!

[pottymouth]

Oh man, your KB must be sticking. Must be looking at too many porn sites with all the extra time you save using VB......... (NOT!!!!)

Re:Developers, developers, developers!

[EddWo]

Windows Server 2003 does this with Volume Shadow Copy.

You can perform daily buckups on network shares and there is even a client for windows that lets the user select which version of the file they want to open. No more support calls for accidentaly deleted files.

http://www.fawcette.com/special/storage/ruest_10 mi n_1/

Corrupt filesystems faster,

[Trigun]

easier and more irrevocably!

Yay!

Re:Corrupt filesystems faster,

[Trigun]

And create the need to upgrade to a Pentium 7! Gotta keep Intel happy and that replacement cycle chugging along.

Re:Corrupt filesystems faster,

[NivenHuH]

Heh.. fastest way to corrupt the filesystem:

drop database winfs;

Re:Corrupt filesystems faster,

[Lord Agni]

Assuming the data that allows you to do the rollback hasn't been corrupted. Is your confidence in MS system software that great?

Re:Corrupt filesystems faster,

[ReelOddeeo]

Everything Microsoft Does can be interpreted as a sneaky new way to enforce DRM.

Some things they do seem to have no other motive than getting your data into their system, and not letting it get back out again. No reason to assume DRM is the motive in all cases.

Re:Corrupt filesystems faster,

[Trigun]

Actually, I am quite versed in the use of MS SQL, and know that it is indeed stable. I also know that poorly written 3rd party apps can fuck it hardcore. Is this the DB's fault? Of course not. Will it ruin your day? If it indexes your porn collection, it certainly will.
I'll let it play with unimportant data like financials, but it ain't getting my porn!

Re:Corrupt filesystems faster,

[fireboy1919]

Don't forget, you also get more availability! So more people will be able to see your data when they decide to browse through the files on your box when they hack it.

Yay!

Re:Corrupt filesystems faster,

[MenTaLguY]

Some things they do seem to have no other motive than getting your data into their system, and not letting it get back out again. No reason to assume DRM is the motive in all cases.

DRM isn't the motive, it's the means.

Re:Corrupt filesystems faster,

[FroMan]

I find your lack of faith... Amusing.
</voice>

Re:Corrupt filesystems faster,

[AKAImBatman]

rollback;

Re:Corrupt filesystems faster,

[sketerpot]

Alright, then truncate he tables in winfs. I don' think you can roll that back, which is why it's so fast.

So basically

[Sir Haxalot]

According to Muglia, the new filesystem will not replace NTFS, but will incorporate feratures of NTFS
It's an 'update' of NTFS, like FAT32 is an 'update' of FAT16.

Re:So basically

[Waffle Iron]

It's an 'update' of NTFS, like FAT32 is an 'update' of FAT16.

Maybe if you're not careful and save your data on an old NTFS partition, you'll end up with something like this:

Properties for file "Proposal 2003-423.doc":
Client: Citywi~1
Project: Networ~1
Status: Pendin~1
Engineer: McMill~1
Site: Minnea~1

Translation:

[Anonymous Coward]

You don't need to buy oracle anymore! ...Just use the windows filesystem!

Re:Translation:

[cgranade]

Correlarry: You don't need to buy SQL Server... MS wouldn't shoot themselves in the foot like that.

Re:Translation:

[ReelOddeeo]

Correlarry: You don't need to buy SQL Server... MS wouldn't shoot themselves in the foot like that.

What if the price of SQL Server is "bundled" into the price of Longhorn? :-)

Or, maybe you really don't need an SQL server. Developers get hooked by an easy to use new way to store and query vast amounts of data. And it is just the filesystem!

Oh, you're going to deploy more than five workloads? You'll a twenty thousand dollar license for that. Sort of like MSDE today. It's free to get you hooked on small time. But need to do something significant, you pay. (Or your client pays.)

Another thing that monopolies do is to give stuff away at a loss to kill competitors. So the grandparent poster could be right. Give away a capable database to kill Oracle. Later, you begin charging money for it. If there is anything that one should learn from watching IBM in the 50's, 60's, 70's is that nothing is more important than market share. Not profit. Not even legality. Even if you are prosecuted and end up paying a huge fine, you still own the market after all competitors are long gone.

It could be called "longhorn" because you're going to get really screwed this time.

Re:Translation:

[cgranade]

It could be called "longhorn" because you're going to get really screwed this time.
That doesn't even bear thinking about... *ouch*

Re:Translation:

[TomV]

I'd guess it'll be the 2006 equivalent of the MSDE at present. It's limited to, IIRC, 5 concurrent users and, in it's present guise, 2GB of data, and isn't supplied with the client tools like Enterprise Manager and ISQLW, but otherwise it's SQLserver2k, and it's bundled with all sorts of Moft apps nowadays - which was one of the reasons Slammer was so effective, of course: hundreds of thousands of people were running SQL Server without even knowing it.

tV

Didn't they buy a ReiserFS license?

[Nicolas MONNET]

Cf. "mystery donators".

Isn't this what Hans Reiser is doing?

[Captain Kirk]

I thought all this was where reiserfs was going?

Re:Isn't this what Hans Reiser is doing?

[Kethinov]

I had a like thought. The article said:

With Longhorn and WinFS, Microsoft is tackling a nagging problem the company has long sought to address. For nearly a decade, the company has touted the vision of a single storage system that would break down barriers between applications and serve up stored information quickly and accurately.

If MS is so concerned about developing a stable and good filesystem, why not just take what's free for the taking the opensource world? I guess doing that would damage their corporate ego too much.

Re:Isn't this what Hans Reiser is doing?

[gfxguy]

The problem is that it would interoperate too freely with other software and operating systems. Can't have that.

Re:Isn't this what Hans Reiser is doing?

[caluml]

The problem is that it would interoperate too freely with other software and operating systems. Can't have that.

No, no :) They could change it slightly, in undocumented ways, like they did with Kerberos and Ldap for Active Directory.

Re:Isn't this what Hans Reiser is doing?

[peter]

> No, no :) They could change it slightly, in
> undocumented ways, like they did with Kerberos and
> Ldap for Active Directory.

Reiserfs is GPLed, so they couldn't do that to the Free version without rewriting from scratch (with the requisite incompatibilities). They could buy a license from namesys and get their own copy of the code that they could gratuitously change, though.

And BTW, the first time I saw something about WinFS, I thought it sounded like reiserfs.

Re:Isn't this what Hans Reiser is doing?

[Kethinov]

Reiserfs is GPLed, so they couldn't do that to the Free version without rewriting from scratch

You sure about that? Redhat is GPLed and it's sold in stores too. What's the difference?

Re:Isn't this what Hans Reiser is doing?

[peter]

MS can't legally distribute GPLed code linked into their kernel without giving away source to the whole thing. RedHat has only the GPL's restrictions on people copying what's on the CDs (i.e. you can copy and redistribute, but you have to make source available), and they give away source, thus fulfilling the requirements of the GPL.

So, in summary, the difference is that RH gives away the source, and I implicitly assumed MS won't.

It's probably...

[Ianoo]

It's probably just a metadata and VFS system built on top of NTFS. By the time Longhorn comes out Reiser4 will have had this for three years.

Re:It's probably...

[lederhosen]

Yes, it is MS SQL Server (a lite version of it)
on top of NTFS, i.e. the filesystem will be the
same.

It is more like a service on top of the filesystem.
I think GNOME is developing something similar.

Re:It's probably...

[sketerpot]

I beleieve you're thinking of GNOME Storage. Looks cool.

the 'holy grail' that is sought by developers.

[3.5 stripes]

They should have a look in the castle ARGGGHHHHHHHHHHHHH, then.

Re:the 'holy grail' that is sought by developers.

[Rupertx]

DEVELOPER: I seek the Grail! I have seen it, here in this operating system! BALMER: No! Oh, no! Bad, bad Bill! DEVELOPER: What is it? BALMER: Oh, wicked, bad, naughty Bill! He told everyone that we have a super-cool operating system which, I just remembered, is grail-shaped. It's not the first time we've had this problem.

Re:the 'holy grail' that is sought by developers.

[Lord Ender]

Maybe he died while carving it?

What about those of us

[Zelet]

who like the filesystem the way it is. I can find any file in my system within 3 or 4 clicks of a mouse because I keep my files organized. How is the new system going to be faster than that? I don't understand how searching for files every time you need them is faster than a file system hierarchy.

Re:What about those of us

[Cyclops]

who like the filesystem the way it is. I can find any file in my system within 3 or 4 clicks of a mouse because I keep my files organized. How is the new system going to be faster than that? I don't understand how searching for files every time you need them is faster than a file system hierarchy.

Maybe not for us, but many people don't have that kind of hierarquical organization, and thus searching is faster.

I would suspect that XML will be used to store metadata from files, and not the files by themselves, and that the SQL database will just be yet another way to store file information, leaving the _real_ files in a normal way.

That said, this is very terrible news. Longhorn is Microsoft's NGSCB (previously known as Palladium, aka Treacherous Computing Platform Alliance software complement to the hardware lockout) and that they're making this huge changes to their file system coupled with Treacherous Computing, that doesn't sound like good news to the world of Free Software.

My only hope is that by the time Longhorn gets here, the world is Free enough so that it flunks.... utterly and completely.

Re:What about those of us

[LostCluster]

I doubt they're going to drop that interface and "logical" way of locating files, but this is going to still make your files appear faster.

See, C:\Foo\Bar1.txt and C:\Foo\Bar2.txt right now do not have to live next to each other on the hard disk. In order to display the contents of C:\Foo\ the system has to search the entire File Allocation Table to look for files that reside in C:\Foo\. This is why a directory that has 1000 entries can sometimes take nearly forever to display in Windows, and even longer if a networking situation is involved.

The basic idea here to replace the simple FAT, or even NTFS, with a database that has a lot of indexes, so that any time you make a request of the file system, the answer is either already ready or very close to being so. So yes, even you should see a speed improvement.

Re:What about those of us

[Dun Malg]

What about those of us who like the filesystem the way it is. I can find any file in my system within 3 or 4 clicks of a mouse because I keep my files organized. How is the new system going to be faster than that? I don't understand how searching for files every time you need them is faster than a file system hierarchy.

Well, a file system is already a database, albeit a fairly poor one. Directories and sub-directories are essentially just fields attached to the record for each file. The new one will still look like the current file systems do, with nested directories and such, because people like it that way. The only difference is, instead of keeping only a few pieces of info in the main directory (filename, size, directory/folder, etc.), there'll be room to add fields. Using these fields, one can create "index" directories containing files with value 'X' in a particular field. It's just an improved way of doing things, not really a different one.

Re:What about those of us

[arcanumas]

I agree.
In fact i would rather have a lightning fast filesystem and use special software to scan and index special data (ID3 tags , Documents info, etc) than have it into the filesystem.
After all, the programs most users use to look up files could very well do such jobs as display Id3 tags and snapshots (eg Nautilus does a good job). Combined with a userspace program that scans and indexes stuff (lets say something "like" GTKtalog ) then you would have a disk that is searchable. Not to mention that if the Userspace program fails you couldn't care less while in a filesystem...
Maybe they could provide a default such program that integrates with Windows Explorer to do the jobs.
Am i missing something? Is there some benefit in a database filesystem that do not understand? (compared to well designed userspace programs)

Re:What about those of us

[lederhosen]

I guess that NTFS works like unix inode based
filesystems, i.e. a directory is a file containing
all the files and subdirs in the dir.

In that case you are wrong. I think you are.
The speed increase will be when you want to list
all your mp3:s. Or all office documents made by
author John.

Re:What about those of us

[quadelirus]

From what I understand, nothing will change in the way data is "stored" from a user standpoint.

C:\myfolder\mydata.data

will remain the same. It's the underpinnings that will change, the way the data is actually physically stored on the disk (disk's do not write the same way your file systems works, in fact they break up files into chunks and throw them all over the disk, that's in part why NTFS needs to be defragged all the time...)

What it will mean, IMHO (and somewhat unknowledgable about fs's), for the end user, is quicker searching of the disk.

I am worried however about the line in the article stating that it "[uses] NTFS...We built on top. NTFS does what it does incredibly well." First off, wasn't Windows ME built 'on top' of previous versions of Windows, and we all know how that turned out... And also NTFS does not do what it does incredibly well, or it wouldn't need to be defragged all the time. Someone needs to disillusion the MS VPs. NTFS does fine, but not outstanding by any stretch of the mind.

I'm curious, what does the new SQL/XML layer mean for applications developers... will it really be that different for us? What if I don't necessarily wan't to use the entirely verbose XML? Or even if I just use [binary encoding] how does adding the XML necessarily make anything better? I use XML where appropriate, mainly for data between apps and web services, but it is by no means an answer to all data storage problems, at least in my opinion.

Re:What about those of us

[_xeno_]

As an AC above me said, you probably should check your facts.

In order to display the contents of C:\Foo\ the system has to search the entire File Allocation Table to look for files that reside in C:\Foo\.

This is wrong. To look up the files in C:\FOO is first looks up the FOO entry on C: off the root drive, and then it loads the FOO directory information directly. No searching the file allocation table, just searching the root directory entry and then the FOO directory entry. With 1000s of files under FAT, then some 1000s of FAT entries do need to be traversed, but definately not the entire table.

NTFS works far more similarly to the way traditional UNIX file systems look up files. It starts with a root node, looks for the directory list of that node, finds the entry it's looking for in that list, and heads off to that node to pick out the list of files in the given directory.

Granted, this is simplifying the matter a bit, but the fact remains that the entire FAT is not searched to look up a single file or even a directory.

The real reason that directories under Windows Explorer take a long time to display is that after Windows Explorer loads the directory it then loads the metadata about the file and in some cases scans the file to check which icon it should display. It's this action that causes the system to go slowly and causes network shares to slow down, not the file system. If you go to the CMD prompt and run a DIR on a given directory, you should notice that it goes by quite speedily with little disk action. Browse to it in Windows Explorer and it could take a while longer depending on the type of files stored in the directory.

Re:What about those of us

[Zan Zu from Eridu]

In order to display the contents of C:\Foo\ the system has to search the entire File Allocation Table to look for files that reside in C:\Foo\.

Nope. You'll have to search the FAT for the clusters in which the directory information is stored. This is similar to walking a single-linked list, if you get cluster numbers that don't belong to the directory (list), your filesystem is corrupt.

The basic idea here to replace the simple FAT, or even NTFS, with a database that has a lot of indexes, so that any time you make a request of the file system, the answer is either already ready or very close to being so. So yes, even you should see a speed improvement.

What is the difference between a directory as-stored-on-disk and a file index? So what happens if you build big indexes that enumerate all files on a disk for quick searching? You have created what comes down to one big directory, containing all files. If you create multiple indexes, that comes down to having files in different directories at the same time; keeping multiple indexes consistent (when files are deleted) will cost overhead, and only speeds up access times if you know in what index to search, if you don't know it will be slower as searching a hierachical file system (because there will be more duplicates, meaning bigger indexes to search).

Re:What about those of us

[novakane007]

How often do people use the search function in the first place?! I've used it once or twice to find finds that users saved in some strage place and couldn't figure out where.

Re:What about those of us

[MasonMcD]

If you've seen OS X Panther's new search capabilities, you'll know how fast things can be: live recursive searching. It's pretty sweet. Works in the finder, as well as Preview for PDFs.

Re:What about those of us

[Smallpond]

I keep my files organized.

A standard file hierarchy is a database with only one dimension, and one hierarchy. You can organize with one first level characteristic, then for each of those subtrees, one 2nd level, and so on. A database lets you have multiple hierarchies simultaneously.

You don't lose organization, you gain multiway organization. The real issue is that you need something better than the typical flat directory listing or tree listing to view the structure of your data. MS needs to combine this with things like the Data Mountain to allow you to visualize in new ways.

Re:What about those of us

[jetmarc]

> In order to display the contents of C:\Foo\ the system has to search the entire
> File Allocation Table to look for files that reside in C:\Foo\. This is why a
> directory that has 1000 entries can sometimes take nearly forever to display in
> Windows, and even longer if a networking situation is involved.

If it were true that searching the whole FAT is consuming so much time, it
wouldn't matter if your open a directory with 1000 entries or _any_ other
directory on the same partition (same FAT!).

As a matter of fact, "huge" directories don't open slowly because of their
layout on the disk. The culprit is the windows file subsystem, which
generates objects et al, for all the files that live in a directory. Even
worse, if you don't just want access to the directory, but _also_ display
the contents in a scrollbox (ie Explorer) - which generates yet another set
of objects. You know, every file needs its icon read, with transpacency,
the datestamp and filetype, the "default" app might be interrogated about
the file, there's a tooltype help to be hooked, etc.

These problems are not cured by adding another layer of complexity. They
will be brought even further to the surface.

Marc

Re:What about those of us

[quadelirus]

Actually NTFS has built in handling of defragmentation but does not entirely block fragmentation... Yes, its better than FAT32 but it also pads 30% of the disk space. If you put 1GB of data on a 1GB NTFS drive 300 MB will be fragmented. Do your homework before calling someone a retard...

Re:What about those of us

[Tackhead]

> The real reason that directories under Windows Explorer take a long time to display is that after Windows Explorer loads the directory it then loads the metadata about the file and in some cases scans the file to check which icon it should display. It's this action that causes the system to go slowly and causes network shares to slow down, not the file system. If you go to the CMD prompt and run a DIR on a given directory, you should notice that it goes by quite speedily with little disk action. Browse to it in Windows Explorer and it could take a while longer depending on the type of files stored in the directory.

Speaking of which, is there a registry key or hidden option I can tweak to make Windoze Exploder NOT DO THIS?

For Fark's sake, I'm also damn near taking my NTFS partitions with my pr0^H^H^Hmp3^H^H^Hware^H^H^H^H uh, application data, and turning them back into FAT32. NTFS = good for the drive that holds the OS. Good for servers. Waste of time for flat data.

Re:What about those of us

[timeOday]

Nope. You'll have to search the FAT for the clusters in which the directory information is stored. This is similar to walking a single-linked list

...and ext2 is much the same. You can improve on this without introducing a whole different filesystem paradigm. See reiserFS, where directory contents are stored in balanced trees instead of a linear datastructure. It doesn't matter much until you get thousands of files into a single directory. People don't do that very often. But Reiser offers that would if our filesystems supported it better; instead, we complicate things by using a database or flat-file database on top of the filesystem when we have lots of little pieces of information to store.

Re:What about those of us

[umeboshi]

id3 is a bad hack to store metadata in the file.
I believe that if there were a semi standard way to store metadata in the fileystem when mp3 applications were first developed, there wouldm't be a need for id3 tags. All things equal though there would probably be versioned metadata schema to keep up with, similiar to id3v1, id3v2, etc. Make the explorer shell plugins easier to write, all sorts of neat things, less wires getting crossed.

Re:What about those of us

[DavidTC]

I don't know in what universe any of this matters, as everyone's FAT is stored in the disk cache 90% of the time.

Re:What about those of us

[timeOday]

I don't know in what universe any of this matters, as everyone's FAT is stored in the disk cache 90% of the time.

I did some benchmarking of ReiserFS against Ext2 for a school paper. Ext2 becomes unworkable at around 10,000 files or so (roughly, as it's a linear increase, not a threshold). At that size, the search time is considerable even without hitting disk. Certainly, if this were a major thorn in peoples' side, it would have been fixed long ago. But fixing it would reduce the need for databases in certain applications (say, the message cache for a usenet reader).

Re:What about those of us

[Brandybuck]

And to think they didn't need WinFS or ReiserFS to do it! Maybe the hiearchical filesystem isn't as obsolete as everyone thinks it is.

Re:What about those of us

[Brandybuck]

Come on man, get a clue! Everyone knows that typing "celebrity pr0n J-lo and Britney" is several orders of magnitude faster than four mouse clicks.

Re:What about those of us

[avdp]

Well, the point of ID3 is that the metadata is encapsulated in the file. Storing that stuff in the filesystem does not make the data (easily) transferable should I move/email/share the file. I suppose that MS would make it easily/seamelessly transferable between Windows machines (if you use SMB to transfer it), but that's about it...

Re:What about those of us

[avdp]

Waste of time for flat data.

Really? The whole point of NTFS (or other Journaling file system) IS to prevent corruption to data. I would argue just the opposite as you: the drive that holds the OS does NOT need to run NTFS (after all, the OS is on a CD you can reinstall at any time, right?) while the data should be on NTFS.

Re:What about those of us

[TomV]

I'm curious, what does the new SQL/XML layer mean for applications developers.

The impression I'm gaining, from keeping up with MSDN and particularly the various Moft bloggers (OPML ) and from reading the article, brings together a few factors:

• The 'Yukon' version of SQL Server will have XML as a native data type, along with the usual varchars and ints and datetimes.
• 'Yukon' will allow the use of XML Schemas to validate the XML data stored is a particular feed.
• WinFS will build on NTFS and SQL Server.
• The cited article mentioned the use of custom XML Schemas to describe the metadata for particular types of data in the file system.
• In 'Yukon' and the 'Whidbey' release of Visual Studio, the Common Language Runtime becomes part of SQL Server itself, so that (and I'm really not at all sure how much I like this with my DBA head on but I love it with my developer head on) it becomes possible to write Stored Procedures in any .net language, there's some sort of automagic mashalling between SQL and Common Type System datatypes, as well as to debug them in Visual Studio.

So, and this is all very fuzzy and hazy for me at the moment, I might as a developer be able to supply an XML Schema which would allow the existing filesystem to both display, index and query my choice of metadata to suit my anticipated use cases. If I use a .net language, I might be able to make winFS queries against the file system natively in whatever language I'm using. Or perhaps several O/FS projects developing similar applications could agree on a Schema for a set of core metadata, which each implementation could extend, to provide compatible metadata via winFS. As I say, very hazy as I'm thinking this up on the fly. There should be an awful lot more information after the PDC as that seems to be the point where a lot of the gags come off about Longhorn.

tV

Re:What about those of us

[Tackhead]

Depends if you're writing frequently to the data. I should have said "static files", not "flat data". For MP3s, once ripped and tagged, the partition could be read-only for all I care :)

Re:What about those of us

[peter]

> If you create multiple indexes, that comes down to having files in different
> directories at the same time; keeping multiple indexes consistent (when files
> are deleted) will cost overhead

Unix filesystems allow files to be linked to multiple directories at once, and they seem to do ok. (inode-based filesystems have a refcount in the inode, which is incremented and decremented as it is linked and unlinked from directories. Now you know why the system call rm(1) uses is unlink(2).). See ln(1) and link(2), and see the second column of the output of ls -l for the number of links a file has.

Unix filesystems don't provide a way to find all links to a file, which is, as you say, what you would need if you wanted to be able to delete a file entirely instead of just unlinking it from its current location. That would cost overhead.
Otherwise, I wouldn't worry.

Re:What about those of us

[umeboshi]

that's what the xml schema would be for.
exporting and importing metadata _without_
actually manipulating the file. moving/emailing/sharing a file would be more like transferrring both the metadata, and actual data a one file. The good thing is with a common metadata handling system, the sender can have a generic metadata manipulator to send the file through for exporting, and the receiver can use a generic metadate parser/translator to import the file.

In a filesystem like reiser4, this becomes a very big win for email as better filters can be made to actually manipulate the email without having to export the darned thing to any number of wheels already made for it. While you have a larger amout of cpu usage compared to a simpler system, cpu ability is increasing much faster than i/o bandwidth, making the tradeoff generally worthwhile.

(sorry if this is somewhat incoherent, i'm getting interrupted too much right now)

Re:What about those of us

[avdp]

I am familiar with XML and XML schemas and what could be done with them, it all sounds really nice in theory but...

With so many file systems in existence today (not even including the ones on portal devices like MP3 players) to expect that the metadata (XML or otherwise) would get transferred from one machine to another is rather naive. Not only that, but this type of transfer of metadata would have to be implemented in email clients, etc as well.

Nice but never gonna happen. And since it won't, ID3 tags are my friend.

Re:What about those of us

[Zan Zu from Eridu]

Unix filesystems don't provide a way to find all links to a file, which is, as you say, what you would need if you wanted to be able to delete a file entirely instead of just unlinking it from its current location. That would cost overhead.

I know about unix hard links, but we're using the indexing to speed up the file searches, and because we're doing away with hierarchical directory structure there is no more need for a tool like hard links to break the hierarchy; so it would be logical (from a MS point of view) to remove the file from all indexes and move it to the "recycled" index when you delete it.

Re:What about those of us

[peter]

delete it from what? If there's no hierarchical directory structure, all you can do is delete it from one particular index (or from all of them, if that's supported). I can imagine a few possible designs, such as a "normal" fs with directories and files, but with indexes as well, or just having indexes (like one big directory).

Re:What about those of us

[bpd1069]

Keep in mind MS OS division usually develops features closely tied to parallel development in the APP division (i.e. MS Office).

With that in mind realize the big marketing hype with Office 2003 is XML support.

Just too bad Longhorn is over 2 years late.

Re:What about those of us

[Zan Zu from Eridu]

The way I interpret this is they're going to index the files in the underlying filesystem not only on name but also on a lot of metadata like content-type, author, creation and access dates, userrights, application group, user-defined properties (like favorites and bookmarks), you name it. If this is the case, it doesn't make sense to remove a file from the metadata indexes but not from the name index; that would defeat the whole concept too, you would end up with one large directory in the name index. (You would be able to remove files from user-defined indexes though.)

Re:What about those of us

[Sabalon]

Your files are organized. Mine are too, but I want to have them organized several different ways.

Let's say I have a copy of Iron Maiden's Wasted Years from the 1987 world tour.

Now...should that be under I for Iron Maiden?
W for Wasted Years?
M for Metal?
80's for when it was?
Bootleg or live since it's not a studio track?

Unless I want to symlink/shortcut/duplicate like mad, I only have one method of placing this file in a hierarchy.

With the new thing, sure the file is in only one place, but with proper meta-data about the file, I can quickly search the database of meta-data to find this on any of the pieces of data.

Re:What about those of us

[EddWo]

Yep thats what I'm getting from it too. It will be interesting to learn a bit more after the PDC.

I don't think weill get a chance to play with it for a while yet. It sounds like it is taking a long time to implement. Probably won't be in a Longhorn beta until 2005.

I quite like the idea of an end-to-end .Net solution. .Net queries and transactions. .Net data access. .Net business logic. .Net web interface.

NTFS + SQL + XML + buzzword compliance?

[Gothmolly]

Why not a little Java thrown in? Or DRM? or TCPA? Or (insert hot new technology here) ?
This seems classic MS-predictive-FUD, where people hold their breath for the Next Release, which is a 1.0 version that sucks. Meanwhile, support people and PHBs have committed to it, so its Too Important To Let Fail. Ultimately, it becomes a time and resource sink the likes of which is only matched by /dev/null, all the while funding MS and driving them to an eventual 3.0 or 4.0 release, which will be Decent, Yet Still Subtly Lacking.
All of this won't help the average user find files easier, and will be massively more bloated and complex (read: too many moving parts, read: Service Pack Hell), and probably REQUIRE that Athlon-64 system we've been drooling over.

Re:NTFS + SQL + XML + buzzword compliance?

[MoxCamel]

Don't forget to throw some .NET in there too.

I'm actually surprised it's not being called .NETFS. I suppose it's because you'd have to use ls -a to see it. :)

Re:NTFS + SQL + XML + buzzword compliance?

[neildiamond]

"Ultimately, it becomes a time and resource sink the likes of which is only matched by /dev/null, all the while funding MS and driving them to an eventual 3.0 or 4.0 release, which will be Decent..."

Wow that's how they made the game Decent? I have to hand it to Microsoft there. If 1.0 is a cruddy filesystem and by 3.0 or so you have a good game, that's what I call innovation.

Re:NTFS + SQL + XML + buzzword compliance?

[mordejai]

Well, that's because after saying '.NET' every two lines for three years, they realized nobody got the idea...

Re:NTFS + SQL + XML + buzzword compliance?

[CAIMLAS]

Sweet virgin Mary! You... are... stupid.

Adjective: decent
Enough to meet a purpose
... and not the name of a popular game series.

Noun: descent
A movement downward
... also the name of a popular game series.

I can't tell whether you were trying to be funny, or trying to be critical; at any rate, you failed on either attempt. The original poster was correct, and you still tried to 'correct' him. Pathetic.

Re:NTFS + SQL + XML + buzzword compliance?

[anonymous loser]

Why not a little Java thrown in? Or DRM? or TCPA?

I'd we willing to wager that TPCA (and indirectly DRM) will be fully supported by this new filesystem. However, they have to introduce it in such a way that consumers don't realize they made a new file system just so they could more fully control the contents of people's hard drives. So, instead they tell you how fast and efficient it is.

Re:NTFS + SQL + XML + buzzword compliance?

[Haeleth]

By the looks of it it was a PUN, i.e. a PLAY ON WORDS, whereby attention was drawn to the similarity of spelling between the unnecessarily capitalised "Decent", and the game "Descent", with humorous intent.

If you're so obsessed with correcting people's spelling that you can't spot a joke when you see one, I'd suggest you need to lighten up a little.

Re:NTFS + SQL + XML + buzzword compliance?

[proj_2501]

it was a horrible pun. he deserves the abuse.

Re:NTFS + SQL + XML + buzzword compliance?

[BurKaZoiD]

If you're so obsessed with correcting people's spelling that you can't spot a joke when you see one, I'd suggest you need to lighten up a little.

1. If it was a pun, it wasn't funny. In order to even perceive it as a pun, you'd have to be just as moronic as the person who wrote it.
2. If YOU are so high-strung you leap at the chance to tell someone else to chill out, perhaps you should take your own advice. This IS the Internet; a context-less, emotion-less void. If there is one thing on the 'Net that pisses me off more than anything, it's when people like you attempt to interpret someone else's intent by reading cold, dead, unfeeling WORDS.

Christ, I've never seen anyone in so dire need of a blowjob....or a life.

Re:NTFS + SQL + XML + buzzword compliance?

[Digital11]

You must be new here... That was an automated spelling/grammar Nazi post. Nothing to see here, move along.

Re:NTFS + SQL + XML + buzzword compliance?

[malakai]

NTFS and SQL are Buzzwords? Is this 1991?

And how exactly is this FUD. Where is the Fear? Where is the doubt? Where's the Uncertainty?

You also say

Meanwhile, support people and PHBs have committed to it, so its Too Important To Let Fail

Wouldn't you mean 'developers' and PHB have committed to it? Support people are always the last to change, and hate it the most.

Look, WinFS is a document repository. It's a good thing if you work with or build document management systems. Because right now, Document Management systems are all over the place. You can use native file systems, with some sort of indexer running, and keep meta-data in a back en relational database, or you can keep meta-data hidden in a non-visible stream (if it's NTFS, and it won't move to another FS), or some people have document management systems where you have "file.doc" and "file.xml". The point is, we're grafting technology together that don't work too well.

WinFS will give us an area where we can dumb BLOB data into it as easily as moving around files, and yet extended that data with specific meta-data to solve business problems.

Don't like it? Don't port it to linux and keep away from it.

My company writes software like this atleast 5 times a year. Normally for lawfirms, generally dealing with a frew hundred thousand files. I for one can't wait for support at an OS level more so than "Use NTFS with MS Indexing Service and a custom Index filter written in C++".

And what is with your FUD:

Ultimately, it becomes a time and resource sink the likes of which is only matched by /dev/null ... All of this won't help the average user find files easier ... massively more bloated and complex

Jeez. That's some serious FUD.

Re:NTFS + SQL + XML + buzzword compliance?

[gl4ss]

look, it is 1991 allover again.

back then they said winnt 4.0 would have all this cool stuff, be object oriented and everything and the kitchen sink, flying windows mapped on '3d' planes, filesystem that does everything, _anything_ they think of 'is going to be in it'. for ms 4 years is eternity of time to switch specs as much as they want, they've done it before and they've already done it with longhorn itself too(and they will keep doing so, anything they say at this point is just what it is: hollow words that taste like candy but are as empty as my wallet).

(yet the drooling masses are only intrested in screenshots, totally oblivious to the fact that they mean nothing, most of the behauvior that has been speculated can already be mimicked on windows with 3rd party tools)

Re:NTFS + SQL + XML + buzzword compliance?

[rutledjw]

This isn't really FUD (Fear Uncertianty Doubt), it's vaporware.

Doesn't exist, won't do what's promised when it does arrive, and what it DOES do it won't do well.

FUD is when MS invites the CIO of my company (and many others) to Redmond to predict the SCO-induced demise of Linux and state that clearly there is one opportunity to escape oblivion - Windows.

THAT'S FUD

Re:NTFS + SQL + XML + buzzword compliance?

[NanoGator]

"The original poster was correct, and you still tried to 'correct' him. Pathetic."

As pathetic as getting Yosemite Sam mad over it?

Thoughts on XML

[I8TheWorm]

I use XML quite a bit in my data-based programs. But I've seen it used WAY too often and in applications where XML just doesn't make any sense (like parsing 1GB files, for instance). XML is a nice tool, but isn't the fastest way to get to data.

That being said, does anyone else think using XML in a filesystem is a horrible way to go? Especially given the hard drive capacity we're seeing today... number of files that can be stored, folders/subfolders, etc...

Unless I misread the article, I just don't see this being a smart move.

Re:Thoughts on XML

[gregarican]

Good point. Things have changed since the days of 640KB ceilings. Back then programmers had to be inventive and ingenious to fit programming tasks under strict hardware limitations. Look at the old video games programmed into 4 KB of space.

Nowadays Microsoft's answer to things is just throw more hardware at a problem. Rather than have lean, mean programming just double the RAM, double the hard drive capacity, double the processing speed, double the bus speed, ad nauseum. Look at some of their recent OS'es. God knows if you could run them on a computer that's more than a year old judging from the minimum requirements list.

The XML observation is true. Parsing through tons of informatation like file system would have isn't the most efficient implemenation for XML. But I'm sure throwing more hardware at the problem will be their answer. Fat, bloated methods doesn't seem to matter to them since hardware prices have dropped so over the last several years.

Re:Thoughts on XML

[_xeno_]

Everything I've heard about XML and WinFS has been so vague that what I'm hoping they mean is the file system information is going to be exported as XML and the underlying system for storing it won't be XML.

This makes an amount of sense - if you want to send a file and maintain the metadata about it, you package it into something that contains the XML file to make it future proof (supposing the format changes, say, because another OS comes too close to being able to read/write WinFS) and the actual data stream. The system on the other end can then restore the XML metadata and then the file data.

Unfortunately, this is wild speculation. I have no idea what they really mean when the say stuff about it being based on a RDBS or XML. So I could be way off - they might be storing XML resource forks for each file or something like that. I have no idea of what they're really doing, but "XML support" could actually be a good idea if it's for data transport and not actually used to store the data on the system. XML was really intended to offer data transport between systems as a kind of Lingua Franca, and this would be a good use of that.

But, as before, this is wild speculation, and we're talking about Microsoft. Every WinFS thing I've heard so far makes it sound like someone's playing Buzzword Bingo. Longhorn is still three years away, so we'll just have to wait for more details as the system actually gets implemented to find out where XML fits into WinFS.

Re:Thoughts on XML

[tomstdenis]

This is just plain wrong. While I agree that windows has bloat around I think you'll find alot of the bloat is in 3rd party or at least non-core applications like Office, etc...

That being said you can certain run WinXP in a resource limited device. My laptop routinely sits at 530Mhz [4x133] while I'm tinkering around in windows and it's just fine. Though my laptop has 240MB of ram free for windows, windows itself only uses about 30MB at most [uses alot for cache]. So you could easily squeeze the core windows and applications written properly in 96MB of ram and a 500Mhz processor. Which last I checked would make such a box about 5 years old thereabouts.

You don't need a 3Ghz 4GB ram box to use windows. You need that to run the 3rd party apps all written in Java/.NET/C++\MFC using the latest raster pexel-pipelining super eyecandy bullshit. Specially games. 9/10 games have no actual gameplay and just eyecandy. Just a sign of the times...

Tom

Re:Thoughts on XML

[TheRaven64]

XML is a nicer way of storing meta-data than traditional DB tables, since you don't have to define the fields before you use them. If it is only used for meta-data (and not for inode tables / FAT) then it shouldn't slow down the system much, as long as the indexing service parses it nicely. I suspect that they are hoping that by the time WinFS is actually ready everyone will have 3GHz+ CPUs, and won't care about the overhead.

Re:Thoughts on XML

[I8TheWorm]

You make two good points. sound like someone's playing Buzzword Bingo and file system information is going to be exported as XML and the underlying system for storing it won't be XML.

One would hope they're planning on using it as an outside repository just for the metadata, but even then, it's doomed to become extremely bloated. Hopefully, it would not be tied directly into file properties, and you would have the option to use it or not at that point

Re:Thoughts on XML

[frank_adrian314159]

I can see it now...

...
<data file="myfile.zqp byteIndex="1">0x2b</data>
...

Boy, am I excited.

Re:Thoughts on XML

[ajs]

This is not a new concept, and it does work out ok.

Your basic linear span of bytes file type becomes a subset of all possible data-structures. Structured file filesystems with various semantics have been around since the 70s (perhaps earlier). XML gives you a way that everyone can agree on to store the schema (of course you'd use a binary XML representation on disk, and have a few prefab schemas (like the DBM type key-value pair) hard-coded into libraries for speed).

This is a good idea, and perhaps one of the few places that I've heard people talk about using XML at a low level that I would agree with in princable.

Of course Microsoft will get it wrong. Because they're idiots? Not at all, there are a lot of bright people there, but Microsoft's priorities are set by their largest customers and for those customers and for marketing reasons, they make some truly AWFUL descisions, like consolodating the Win32 API and throwing away the multi-tiered, user-space service approach that NT was originally supposed to have on top of HAL and the NT microkernel. That was done because MS saw a need to give their largest base of developers (corporate in-house mostly) as close to a seemless transition from Win3.1 as possible, and for no real technical reason that had to do with NT.

That ruined what was likely to to have been one of the coolest pieces of software that Microsoft would have ever produced. NT (now the heart of XP and Longhorn) is still a cool OS at its core, but as I expect to happen with this filesystem, it was so hobbled by the needs of their business customers that it took a decade to extract any real value from that.

I'm not MS-bashing. I'm a Linux/UNIX/BSD user, but I'm willing to accept that good developers work on all sorts of software, open and proprietary. The problem is that the larger a software business is, the less voice the developers have.

Personally, I'm waiting to see where Reiser goes...

Re:Thoughts on XML

[I8TheWorm]

Then run 3 instances of IE on top of that (since we're talking about MS products). 23MB per IE, that's an additional 69MB. And in checking out my task manager, explorer.exe is taking 12MB, services.exe 18MB, svchost.exe a combined 13MB, outlook.exe 6MB.

I've ignored quite a few system critical process in win2k (lsass, mdm, regsvc, system, winlogon, etc..) but you get the drift. Windows in low RAM is fine if you only want an app or two open at a time. MS has a history of using more resources than competitors do in just about any app/os.

The discussion though was about XML and poor uses of it. And storing metadata for thousands of files is probably not a good use.

Re:Thoughts on XML

[I8TheWorm]

Very well thought out. I wonder, though, why not go with an ISO standard flatfile, which will be smaller and thus much faster to parse?

Re:Thoughts on XML

[oobar]

I think this whole XML thing is a red herring. I think what they mean by "incorporate features of XML" is that the filesystem will support an extensible set of metadata fields for each file. So where currently you have creation time, modification time, security attributes, etc. in the new FS you will be able to define and store any arbitrary metadata attribute for a file... For example, take images: you could enter a description of what's in the picture and it would be stored with the file. These attributes might even be described by an XML schema and stored in an XML format, but that doesn't mean the filesystem uses XML for its underlying structure. You could imagine the case of having "mypicture.jpg" and "mypicture-metadata.xml", where the latter contains the metadata for the former... and instead of storing this as two files, the filesystem keeps track of the two under a single "entry" in the filesystem. So when you want to access the payload, or the main binary data, it would be handled the same as NTFS does now. But the XML metadata is also indexed somewhere, so when you want to search the description tag for some keyword, it will also be able to find mypicture.jpg. You could probably hack something like this together with NTFS, by using the "alternate streams" or whatever it's called that lets you store extra/alternative file data in the filesystem.

I don't think anyone from MS has ever said that they are thinking of using XML as the actual underlying engine of the filesystem. That would be really stupid. Just think of this as taking the current existing "file attributes" and storing them in XML alongside the regular file's contents.

Re:Thoughts on XML

[gregarican]

Whatever, dude. If you think each version of Microsoft's OS doesn't have exponentially higher hardware requirements you are mistaken. My company has about half of its workstations with PIII 550/600 MHz CPU's and 64 MB RAM. Upgrading a couple of these from Windows 98 to Windows 2000 has left them noticeably slower. Of course this isn't news in any event. And these PC's are over three years old after all. But Linux in comparison obviously can run better on older hardware. We all know this.

The *point* of my post was how typical Microsoft's software development is. It is mostly engineering with the arrogant assumption that its intended audience all have PC's a year old or newer. It's ridiculous. Look at all of the .NET Framework crap. I was forced to install version 1.0 on my workstation just to update some Great P(l)ains tax tables. All the updates did was just execute a handful of lines of SQL scripts. But I had to download over 130 MB of .NET crap to get them.

Micro$oft should be on a VH-1 "Behind the Music" segment. How they were hungry and struggling to make it, then turning out some of their best stuff. Then once they made it, how complacent and arrogant they became. Assuming that the public adored them no matter what they did. Finally, how they tried to win back their fans by getting hungry again. Returning to their roots. In a similar fashion they should strip away all of their layers of complication and revert back to Windows 3.1 and Foxpro (j/k).

Re:Thoughts on XML

[Citizen of Earth]

I use XML quite a bit in my data-based programs. But I've seen it used WAY too often and in applications where XML just doesn't make any sense (like parsing 1GB files, for instance). XML is a nice tool, but isn't the fastest way to get to data.

Employ a binary-encoding approach. Use embedded indexes. Get binary performance and XML flexibility. W3C is pondering the subject of binary encoding.

Re:Thoughts on XML

[I8TheWorm]

I think you're right. My thoughts on that, though, were still related to a bloated filesystem, in that the XML doc, unless each file or folder has a separate doc, will get bloated, and take a great amount of time to parse. I have two 40GB HD's on my main PC at home, and probably have a combined 15GB of space available. I wouldn't venture a guess as to how many files that is, but certainly in the thousands.

While XML is a nice standard and easy to read/parse, it's filled with bloat, by definition. Rather than know by some standard which character position begins the field-def, you have tags/closing tags that take up more space in the file. I can imagine that MS wouldn't think of storing all metadata in one large file (given the fiasco that the regstry has turned into). So how would they tackle the problem? Probably a separate file for each folder, not unlike how folder properties are stored now. However, with each file move, now you would have to change the XML doc in two folders, as well as change the header of the file to show where it's stored now. Now imagine doing that for 5000 files. Pretty daunting task, given the additional work of checking tags, possibly against a DTD.

It's just my personal opinion that XML is already overused for tasks it wasn't intended for, and this might be another. Maybe looking into an OODBS structured filesystem would be time better spent. Who knows....

Re:Thoughts on XML

[I8TheWorm]

Both of those are new to me, and seem pretty interesting. Thanks for the links!

Re:Thoughts on XML

[Minna Kirai]

XML already exists. It has been defined already, and binary-encoding is not part of it.

There should always have been binary-encoding options as part of the XML specification, but now it's rather too late. XML parsers from many sources have already been deployed around the world.

Design shortcomings become expontentially harder to correct the longer a software project has gone on. By the time it's deployed for real systems in the wild, fundamental changes would be almost prohibitively disruptive.

Re:Thoughts on XML

[Per Wigren]

Thanks for the links! BXML was new to me but it seems very similar to EBML which is used and developed by the Matroska-team. I wonder what their pros/cons are...

Re:Thoughts on XML

[g0at]

princable

Is that the name of the thing that connects my Laserjet to my computer? :P

-ben

Re:Thoughts on XML

[Ed Avis]

It would be insane to represent the filesystem on disk as a big XML document. It makes a lot of sense to let you view particular portions of the filesystem as XML, or to give it XML-like semantics (ordered elements, unordered attributes, elements can contain text mixed with other elements) and adapt existing query languages (XQL etc etc) to work with it.

Re:Thoughts on XML

[I8TheWorm]

Oh, I understand that. The problem I have with it is the number of files you would have to represent. How do you overcome that? With a separate XML file at either the folder or the file level. But that adds yet another layer that the os has to handle on file moves, creates, deletes, etc... Just another speed bump in the process, as far as I can tell.

Re:Thoughts on XML

[Ed Avis]

You don't have to represent the filesystem as XML on disk at all. You'd probably store it in some performance-optimized binary format as at present, and generate XML when it's wanted.

Re:Thoughts on XML

[FreshFunk510]

We all know the benefits of XML, but the question is how does this benefit us at the personal filesystem level? Here's my guess:

The article talked about how they were integrating web services largely into Longhorn and how it would be an integral part. They also talked about the ability to easy replicate data (and hard drive) remotely. By having the relational information stored as XML, you face little opposition translating this information into a webservice.

So, for example, let's say I had a bunch of business files I was working on at my home comp. Now let's say I want to easily replicate those directories at my work place, on my laptop, at teh client site, and several other spots. Sure, I coudl write a whole app to do it now, but if that's seamlessly done through file system and web services it provides wide availability integrated into an OS (assuming I'm running those same OSs on the other machines).

Sure it coudl be done with other technologies but the point is that if you create this low-level infrastructure you enable this sort of sharing and syncing technology across the board. You could simply do the same with media files, etc.

Re:Thoughts on XML

[Citizen of Earth]

Both of those are new to me, and seem pretty interesting. Thanks for the links!

Here are some additional rough ideas to make the encoding even more compact. Content & tokens are collapsed for compactness. The byte-oriented macro idea makes the encoding just as compact as schema-based approaches (factoring out fixed structuring) without the complexity of schema processing. It is also designed to facilitate byte-for-byte-identical round trips between XML and BXML encoding.

You could define a macro so that foolishly overstructured data such as the following can be represented with a raw byte array and still be processed generally or with a special reader that recognizes the macro signature and reads only the raw array:

<Scanline>
<Pixel>
<Red>111</Red>
<Green>222</Green>
<Blue>123</Blue>
</Pixel> ...

Define a macro for the fixed parts of <Pixel> with single-byte instance-data references for the color components and then a 1024-pixel scanline can be represented with a macro-reference token byte, macro id byte, count of 1024 (two bytes), and 3*1024 raw bytes of sample values. The macro is replayed 1024 times filling in the instance-data bytes where appropriate. General records work the same way, with fixed and variable-length fields of instance data.

Re:Thoughts on XML

[MattRog]

I really hope they're actually storing the meta-data in a DBMS table and only using XML as an interface when some DLL function 'get_xml_metadata' is called.

Re:Thoughts on XML

[itsari]

That being said, does anyone else think using XML in a filesystem is a horrible way to go?

Maybe not. XML is structured in a similer way as a filesystem heirarchy, which could probably prove useful.

Re:Thoughts on XML

[EddWo]

You have to consider the cost in these things.

At some point the cost of hardware fell below the cost of a programmer. Is it better to pay someone to tweak and optimise code for a couple of months to reduce memory footprint, at the same time risking creating new bugs, or just buy a 256mb stick for $30?

As programmers we all want to try to write tight efficient code but businesses would rather have features, stability and maintainability.

Thats where Java and .Net come in. Rapid development, high level OOP design, automatic memory management etc. It may not be efficient but it works and if you want more speed throw more hardware at it. As long as the cost of hardware keeps falling then code bloat is the way things will go.

This doesn't work so well with critical sections of the OS kernel. That is mostly still written in C but it requires intensive testing for the smallest modifications.

Also Microsoft doesn't target new OS releases at old hardware. Look at the problems trying to run win2k on non acpi compliant systems. Most sales come from OEMs with their Windows Logo Certified boxes. They still sell upgrades for home users but they would prefer not to because they don't want people calling them for support. Look at product activation, a non issue for OEM boxes as the copy of the OS is tied to the BIOS.
If Longhorn or the version after that requires TCPA to function you may find you can't just buy it off the shelf. The only way to get the TabletPC and Media Centre varients is to get them with hardware.

Re:Thoughts on XML

[FreshFunk510]

RTFA

gnome-storage

[joejg]

This is an interesting turn of events. When discussing the gnome-storage project with people, they mentioned that M$ would be incorporating WinFS as an actual file system. And they wondered how the gnome-storage dev's were going to create a database upon a filesystem. But as it turns out M$ isn't making a new file system and for all we know they are lagging behind the great work done by Seth Nickell and crew.

Tom's hardware

[Sir Haxalot]

has a similar artical, but with the Tom's Hardware you have this quote: 'He indicated that there are also plans to include Win FS in the Windows Server 2003 generation.'
This certainly interested me, because if it's any good, this could completley change the Server market, providing much need competition.

WebDAV???

[stonebeat.org]

Sounds like WebDAV + Catacomb to me.

Do you guys really think

[rhinoX]

that the NT and NTFS teams would allow the SQL Server group to take such a huge chunk of control from them? You've got to be kidding me! The filesystem will not be anything new. The SQL guys at MS have been trying to move to a DB FS for a while, but let's face it - the performance will absolutely SUCK. At most, this is SQL server for metadata slapped "on top of" NTFS, period.

Uh huh

[xaoslaad]

According to Muglia, the new filesystem will not replace NTFS, but will incorporate feratures of NTFS, SQL, and XML all into a filesystem which, accoring to Microsoft, will open up a whole new world of information availability

Yep, the first SQL vulnerbaility after WinFS will make the whole disk available rather that just the database.

(joking aside it does seem interesting...)

great

[pizza_milkshake]

...an XML-based filesystem; i can imagine the speed now

Single storage system.

[Interruach]

For nearly a decade, the company has touted the vision of a single storage system that would break down barriers between applications and serve up stored information quickly and accurately

So they're just kicking themselves that they haven't moved from drive letters to a unix style mount-anywhere style filesystem?
If they've wanted that for 10 years, they should have done it in '95

Re:Single storage system.

[xswl0931]

On XP, you can mount filesystems anywhere, not just restricted to drive letters anymore. mountvol.exe /?

Re:Single storage system.

[ironygranny]

Try deleting a file from a mounted volume in windows explorer. It won't work. It has to do with the fact that Windows creates a seperate "Recycle Bin" for each volume, and can't figure out where to send things when the volume is mounted to a folder on another volume. This isn't such a big deal for the server market, it's just an annoyance for dorks like me with nothing better to do than kludge a unix-like structure into windows.

Does this mean that Bill Gates represents ...

[burgburgburg]

Merlin: "Uther, say the words."

Uther: "ONE LAND, ONE KING!"

Cool

[Pedrito]

Look, I hate to say it, but I've kind of been looking for this capability in a file system. So much so that recently I had been thinking of writing software to do something along these lines.

In particular what I'm looking for (and maybe other people are too), is a file system where it's easy to find files. I don't mean finding them by the name, but by content, and not just text, but graphics, executeable, you name it. For example, I have tons of pictures from my digital camera, scattered into different directories, on different drives. I'd like to be able to query for example, "photo me sophie" and get back all pictures of Sophie and myself.

Now, admittedly, this would also add on some responsibility to tag keywords to the files, and I've thought of ways of doing that as well (for example, applying keywords associated with a directory automatically to files placed in the directory).

I haven't worked out all the kinks, but to me, being able to find stuff quickly in file systems that are continually growing would be a huge bonus.

Re:Cool

[68K]

Have you considered organising your photo collection? If you're prepared to tag your photographs, why not just rename them to something more readable and stick 'em in a directory structure that make sense?

I've got thousands of photographs (about 5Gb's worth) taken over the years organised into a reasonably efficient directory structure, and I didn't have to write any software to do it. :-)

Re:Cool

[nolife]

I have tons of pictures from my digital camera, scattered into different directories, on different drives.

Here's something you can do and it works on any platform. Don't scatter them? Is it really that hard to have a pictures directory and give the subdirectories descriptive names?

My mp3's go into soundtracks, remixes, regular, comedy, kids etc..
My downloaded files go into internet, drivers, emu, patch, tools, serialz etc..
My scanned records and receipts go into 2003/taxes, cars, house, medical, general
My camera pics go into 2003/house, kids, vacation, cars, xmas etc..

I know where they are and when they've been backed up. I do not want to rely on a search term (other then *.jpg or something similar) to verify I found them all or have everything that may or may not apply.

Re:Cool

[bhoult]

Have been thinking of makeing something like this actually. Was just going to make something to replace the locate command that stores complete file info + thumbnails + keywords in a postgres db and make a front end for the command line and also a web-based cgi.

Should be fairly simple and can be extended for more complex queries.

Re:Cool

[Demandred]

Check out A Logic Filesystem from this year's Usenix conference. It is a filesystem with precisely these capabilities.

Re:Cool

[penguin7of9]

There are plenty of tools for indexing and searching your disk; that stuff shouldn't go into fhe file system.

I haven't worked out all the kinks, but to me, being able to find stuff quickly in file systems that are continually growing would be a huge bonus.

Well, duh: that's what UNIX is all about. That's why you have dozens of command line tools to search and manipulate files with. For example, to find all TeX files in your home directory containing a keyword, use this:

$ locate .tex | grep $HOME | xargs grep -l keyword

For finding images, many of the image browsing applications for Linux now include search as well, both by keywords and by appearance.

he said,

[CAIMLAS]

"NTFS does what it does incredibly well."

Which is what, exactly? In my experience, NTFS has been nothing but slow, unreliable, and utterly unscaleable. Ever try running an NTFS volume as a cache or some other-small file storage device? Utterly unuseable.

So if WinFS is to augment NTFS, my prayers go out to you Windows administrators.

Re:he said,

[penguin7of9]

Which is what, exactly?

Easy:

• Require frequent hardware upgrades to make up for its inefficiencies.
  
• Keep competitors and OSS from being able to access data on Windows partitions reliably.
  
• Keep software companies selling backup and file system maintenance software in business.
  

It is really good at all of those things.

Re:he said,

[Per Wigren]

• Making more powerful hardware available for lower prices improving performance even more for those of us who use more efficient OSes.

feh

[syle]

More like the holy grail of buzzwords.

When was the last time...

[1WingedAngel]

your 'holy grail' turned out to be filled with urine?

New Filing Cabinet System Announced

[doppleganger871]

According to sources close to industry experts, a new filing cabinet will be brought to market sometime in the next year or two. Instead of having files, labeled with what their contents are, you will have a master file cabinet, with thousands of records that will give you a map on how to open a drawer and decipher a complex set of instructions to find the paper you're looking for. The system will revolutionize the way papers are stored in folders. Previously, there was no large, cryptic system for shuffling papers around, now there is a standard way to misplace items. No longer will you have to look just in drawers, but you may not be able to even find the cryptic instructions to lead you to the drawer to start looking in. Details are sketchy, but some have eluded to possible bookworm attacks, that could corrupt the cryptography and therefore render your whole filing cabinet useless. Industry experts suggest that an anti-bookworm product could be available shortly to help protect from this.

Re:New Filing Cabinet System Announced

[zasos]

now that's funny AND insightfull!!!! mod the parent up!

Re:New Filing Cabinet System Announced

[Confessed Geek]

You do realize you just described the dewey decimal system and a card catalog, Right?

Re:New Filing Cabinet System Announced

[doppleganger871]

The whole thing wasn't meant to be totally accurate, i was just typing what my brain was spewing on the fly, I didn't even go back to proofread it. Just an example of things that happen when you're in a M$ shop all day.

Virus this, crash that, corrupted this, not working that, etc.

Glad someone found some humor in it, tho :)

Holy Grail??!?!?!

[kldavis4]

As a developer I am a bit surprised by this statement? I don't believe I am seeking any such holy grail. What is he talking about? Are there any programmers out there who actually believe they might benefit from such a thing?

Why risk it?

[Ars-Fartsica]

While I applaud new thinking from any OS vendor, this type of re-architecting will mean little to the core home/business user. Are files really that hard to find on your computer? Are people really ready to throw their data into a virtual bucket and use pattern matching to find it later? Maybe, but consider the risks:

1. It is discivered to be broken shortly after release. This would cripple the entire OS and require a huge mae culpa from MS.

2. It is insecure. Once again, the whole OS would be undermined, users would revolt.

3. No one "gets it". The R&D cost and time spent will be for naught, people will keep on thinking like they always have - organizing files through Windows Explorer.

Considering the huge installed base and relatively low requirements of MS users (cruise the web, listen to music, read email), it just doesn't seem worth the risk. I would offer that secure, reliable mediocrity is what they should shoot for, they already own the market for desktop OSs and can't possibly convert the 5% who use Apple/linux etc.

Re:Why risk it?

[tesmako]

While I applaud new thinking from any OS vendor, this type of re-architecting will mean little to the core MS-DOS user. Do we really need to run more than one program at once? Are people really ready to leave the control of their computer up to a scheduler and virtual memory system to manage? Maybe, but consider the risks:

1. It is discovered to be broken shortly after release. This would cripple the entire OS and require a huge mae culpa from MS.

2. It is insecure. Once again, the whole OS would be undermined, users would revolt.

3. No one "gets it". The R&D cost and time spent will be for naught, people will keep on thinking like they always have - I am working in this program so I will run only this program.

Considering the huge installed base of MS-DOS and relatively low requirements of MS users (Lotus123, Commander Keen), it just doesn't seem worth the risk. I would offer that secure, reliable mediocrity is what they should shoot for, they already own the market for desktop OSs and can't possibly convert the 35% who use Apple/UNIX/AmigaOS etc.

SQL, XML not a Holy Grail: relational would be.

[leandrod]

SQL is just a corruption of the relational model, loosing power and simplicity. XML is just markup, and perhaps a nice programming convenience, but attempts to force feed it into databases and filesystems are bound to fail as miserably and costly as the similar attempts to force feed OO.

On the other hand, even SQL would be better than the current hierarchical file systems. And a truly relational database system such as Alphora Dataphor as a filesystem would be my technical Holy Grail, yes -- just not in MS-WNT. Make that work in GNU and I'd be happy as happy can be.

Re:SQL, XML not a Holy Grail: relational would be.

[Prowl]

Care to elaborate on the "corruption of the relational model" bit? I know little of the relational calculus, but always assumed SQL was just a way of expressing it.

Please reply, I'm really intruiged :-)

Re:SQL, XML not a Holy Grail: relational would be.

[leandrod]

> Care to elaborate on the "corruption of the relational model" bit?

The best would be to read the latest editions of Date's and Pascal's books, but just to sum it up:

Tables are not relations: they may contain duplicates and NULLs, thus breaking the power of and complicating significantly relation 2-valued logic and relational operators.

Lack of view updateability: thus lacking data independence.

Lack of user-defined types: thus making people think they need OODBMSs.

But don't take my word for it. If you can't get the books, some good arguments are delineated at http://dmoz.org./Computers/Software/Databases/Rela tional/

> I know little of the relational calculus, but always assumed SQL was just a way of expressing it.

Actually SQL was just a half-baked prototype gone awry, and it is a mix of calculus and algebra to boot.

Re:SQL, XML not a Holy Grail: relational would be.

[Prowl]

Very enlightening. Thanks for the reply.

Your sig says your a DBA. What's your prefered RDBMS software? We use MySQL/InnoDB, although that doesn't come across too favourably in some of the articles you point to (although for that matter, neither do any others...)

Are there any implementations that don't use SQL?

Re:SQL, XML not a Holy Grail: relational would be.

[leandrod]

> Thanks for the reply.

You're welcome!

> Your sig says your a DBA.

At least when I'm employed, which is not the case now... and to tell the truth, my last job wasn't as a DBA as well. But I am trying to return.

> What's your prefered RDBMS software?

At the moment there are none. The best free SQL DBMS at the moment seems to be PostgreSQL, and the proprietary one IBM DB/2.

> We use MySQL/InnoDB, although that doesn't come across too favourably in some of the articles you point to (although for that matter, neither do any others...)

InnoDB is as good a DB engine as you could wish, but MySQL is a complete disaster. Seems like its SQL dialect is even less compliant than Oracle's; it is actually misnomed at that. In fact it is so bad that the upgrade path is SAPdb, which is at Oracle v7 level!

> Are there any implementations that don't use SQL?

Yes, but none yet a full RDBMS.

Ingres, which used QUEL, isn't anymore maintained, and that's a big loss.

Alphora Dataphor is the one closest to the Holy Grail, but it is more of a rules engine that translates a proper relational language to SQL, thus precluding the proper otimisation. Moreoever, it is .Net. The authors are trying to port it either by Mono or by C recoding, but they haven't yet found their free software strategy. Couple it with IBM DB/2 or PostgreSQL on some kind of POSIX platform, and you're on to something.

Then there are incipient libraries like Opus, Duro... they are bound to eventually reach D compliance, be it D&D's Tutorial D, Alphora's Dataphor D4 compatibility or something else. But this will likely take years.

It's a simple question of weight ratios...

[davidstrauss]

Well, it isn't, but the overhead of such an inplementation has to be balanced with performance. NTFS is already fairly high overhead. I would rather see a small NTFS OS loader partition that initializes a main database partition. That way, there wouldn't be the overhead of NTFS on the main partition. I would hate to have my media library stored fully in NTFS and described by a database. It would be no different than using a modern media player.

Re:It's a simple question of weight ratios...

[NaugaHunter]

So what you're saying is that a five ounce database (MSSQL) can not carry a one pound filesystem (NTFS)?

BuzzwordFS

[Ender Ryan]

In other news today, Microsoft announced the successor to WinFS, which is slated to appear in Windows LT(Long Tool) in 2012, BuzzwordFS. BuzzwordFS will not replace WinFS, it will incorporate features of WinFS, as well as every other possible TLA used in the computer industry, for complete buzzword compliance.

A few points

[arvindn]

• Beos poineered the FS as a database concept a long time back
• ReiserFS already does these things (incorporating metadata into the FS), and does it extremely well (performance wise)
• If Longhorn implements an intelligent FS it would be a good thing for *nix users as well who would otherwise have a problem transferring files/archives to/from windows machines.
• The big question is not technical feasibility and not even backward compatibility. Its how to avoid chaos among developers. FS with metadata is such a leap ahead of conventional FSes that a lot of code will have to be rewritten; many programs will have to change their architecture in fundamental ways; in the transition period two trees of code will probably need to be maintained, one for the new and one for the old FSs.
• We (the OSS community) need to work on a standard API for next gen FSs. Otherwise there would be a whole bunch of similar but mutualy incompatible FSs.

Re:A few points

[tuffy]

Right now, filesystems store data as big blocks of bytes with some extra metadata on the side. Even Unix-likes have UIDs, permission bits and so forth. Storing arbitrary metadata or RDBM-style data is an interesting progression, but you're screwed once you try and transfer that data elsewhere over typical channels - like a simple HTTP transfer.

The typical solution is to use OS-specific archiving tools (like tar) to handle the transfer of that metadata. But, for example, would you really want your audio data stored in OS-specific tags? How would that metadata be transferred over the web? Would you really want to risk losing it when trying to transfer to a portable device, or an older OS?

I think abitrary metadata in the filesystem would be a great place to cache actual metadata stored in a file's big chunk of bytes, but I don't think it'll ever become ubiquitous the way Microsoft envisions. Since even if they should make it standard in a new Windows version, there'll be far too many other systems (and older Windows versions) that'll be too incompatible to cope.

Re:A few points

[WuphonsReach]

There was a similar problem back in the MS-DOS / OS/2 / Win95 days. MS-DOS only supported 8.3 filenames, OS/2 and Win95 supported long filenames - but differed in how they stored them in the FAT tables.

End result was a lot of files named SPREAD~1.WK3...

MetaData belongs in the file, not in the file-system.

I don't think I could stand...

[sniser3]

... 3 years of "bla bla random feature of longhorn to keep microsoft in the press bla bla". Lots of exciting stuff is availabla right now, in 3 years a lot will have happened, let's ignore MS a bit shall we.

Complete this sentence:

[ivan256]

He goes on to describe such a filesystem as the 'holy grail' that is sought by developers... ...of high end processors, memory manufacturers, and name brand PC makers, who's sales have been down lately due to current software running well enough on previous generation hardware.

Re:Complete this sentence:

[cK-Gunslinger]

Very true. Hardware and software developers have a very symbiotic relationship. It use to be games that drove PC hardware sales, but that's started to change recently. High-end equipment from over a year ago is *still* fast enough to run the latest Windows OS as well as the most demanding games out today. (An AMD XP 2000 w/Radeon 9700 is still overkill for 75% of new games.)

Someone has to pick up the slack and MS seems to have grabbed it with both hands (they already had the right hand on it.)

Re:Complete this sentence:

[hawkestein]

id Software might step up to the plate, if Doom III ever comes out. That game should eat some serious CPU and GPU cycles.

Re:Complete this sentence:

[cK-Gunslinger]

True, as well as HL2, but they've both been pushed off to next year. It's still very unusual that year-old hardware is doing so well. Sure, you can upgrade your 9700 to a 9800 XT and your P4 2.8GHZ to a P4 3.2GHz E or whatever, but what does that gain you? Your Quake 3 FPS jump from 340 to 380? While a 40 FPS increase used to be nothing to sneeze at, it's not as meaningful when it only represents a 12% increase. There are no tangible benefits. No games go from being nearly-unplayable to enjoyable.

*NIX could do a similar thing..

[CooCooCaChoo]

It would require Oracle DB as part of the equation.

The operating system core resides on an un-DB partition which has the basic core plus the Oracle DB engine, the other part is a raw partition. Instead of the information being a partition, the DB can be mounted as a partition in the traditional sense so that to the end user, they will notice no difference. All they will see when they go df is a piddly 200MB partition and the rest mounted under something else.

In otherwords, it is nothing revolutionary. Oracle was promoting this years ago when Larry claimed that "operating systems are dead".

Re:*NIX could do a similar thing..

[Isaac-Lew]

It would require Oracle DB as part of the equation.

Why couldn't mysql, postgresql or another open source DB be used? Last time I checked, Oracle was expensive.

Price of admission is a full disk comprimise

[Ars-Fartsica]

Yes it is cool, but in version 1 you have to take on an enormous risk - virii, bugs, other issues crippling your entire OS is a more sophisiticated way than you can conceive a defense against. The more complex the system the more complex the attack. MS users do not require such complex systems - I don't know why MS is risking exposing them to more sophisiticated attacks.

Wrong mythology

[Soulfader]

"Holy Grail?" Maybe Pandora's Box would be more apt.

Re:Wrong mythology

[Bigby]

But you must remember that this is the same company that said XP is "the best thing since sliced bread".

Urgh... Help!

[jo42]

Er, ahm, I just don't get it... Why would I want a filesystem to be like this? How would it "open up a whole new world of information availability"? Mebbe me wee brain doesn't get pablum-based marketing speak. Someone please clarify... How would, or why would, this change the way I store and retrieve my documents, photos, etc.? Would the computer somehow read my mind and keep things organized for me? Or would I still have to type in document, photo, etc. names and keep them organized in categories like Proposals, Invoices, Grand Canyon Rafting Trip, etc.? I already do this using folders, etc. Or is this all much like eye candy that the Longhorn (cowpies!) UI is supposed to have?

Re:Urgh... Help!

[tomcio.s]

Yes. I agree, but what they are striving to do is to priovide a service where let's say you make a proposal document, then later on you reference it from another document. Now the proposal can easily be copied over and stored in 2 places, but why not just give it a sim-link instead. Wait. That is not part of the windows file system right now.
So they are solving it by keying relevant documents in a file/folder based filesystem to relational database, that will let you find data faster (i.e. I remember I did this for this project, but can't seem to rember where I did it first. So the db will store the connections and let you know). It is also worthy of noting that it will allow you to keep one copy of the document, such that you only have one place to update. Ever.

In any case, someone kick me for saying this, but M$ is going the right way here, albeit round about anywho. The filesystems do need an overhoul big time. Especially as the amount of information stored increases.

Theory only?

[countach]

This is a great idea in theory, but the fact is for most people simple == good enough. This thing is sure to be a real tricky thing to use at first, and it will take ages for apps to support it probably. For most people it will be pure bloatware.

On the other hand, maybe Microsoft will pull a beauty out of the hat.... Naaahhhhhhh.

M$'s real holy grail

[smartin]

And of course WinFS will provide them with the ability to enforce DRM down to the file system level. Which is the Holy Grail of squeezing every last penny out of your customers, and making exclusive deals with evil associations.

Three words

[sonofasailor]

Trust worthy computing lets go over to cert and count with me then hook this b$%ch into whatever iteration of directory services they come up with I thought M$ was promising more security not less maybe the lawsuits will help was this the massive overhaul of the OS M$ was refering to? Hey Bill FIX the FSCKING kernel!!

MS Filesystems are Internet Enabled!!!!

[zulux]

In 1997 Microsoft made a pledge to Internet Enable all of their tehcnologies - and true to their word, even their file-systems are now internet enabled.

It's really easy to administer. Just plug you Windows computer into the internet, wait for a few minuits for a helpfull worm - and PRESTO!!!

You file system is on the Internet Baby!!!

WinFS is an object oriented database

[category9]

WinFS sounds like it is going to be an object oriented database mounted as a file system. It will have standard filesystem object types such as directories and files, but developers will be able to expand on these basic types to create complex media types and intergrate databases with the file system.

I can't wait, although i don't know what its going to do to my business. It looks like its going to do what our object oriented database does plus more. We better start coding catchup. Maybe if we release a similar product to WinFS in say a year we can corner that market before Microsoft does. Wishfull thinking.
I don't want to go the same way at Netscape...

---
Phil Harvey
construct software
object oriented database solutions

*ahem* Bullshit

[sik0fewl]

"Today, applications encapsulate data. In the future, applications will be able to read and write data created with multiple applications," Muglia said. "Information opens up dramatically."

*ahem* Bullshit. Unless you're trying to open a Microsoft document with a Microsoft application I'm thinking you're gonna have a lot of trouble. And of course when he said "applications encapsulate data" he was obviously only referring to Windows applications.

Re:*ahem* Bullshit

[Garwulf]

Actually, there's already plenty of applications that can open documents created by other applications. With the latest WordPerfect I can open up Word files, and I imagine that Open Office is similar in compatibility.

What blows my mind is that if you take a close look at the statement, it's basically describing the situation between applications that has existed for the last decade and a half as though it is something new and amazing. After all, the only way a good desktop app. can be competitive now is to be interoperable with everybody else (even Microsoft tries to do it with Word).

Holy grail?

[Eudial]

"He goes on to describe such a filesystem as the 'holy grail' that is sought by developers"

And i thought AI was the holy grail! Damnit!

not your database

[happyfrogcow]

I assume MS will want this database to be a machine/group of machines under their control, someplace on their network, where they control "your" data.

But...I like NTFS!

[NineNine]

NTFS has been an excellent FS for years! It's fast, it supports massive drives and massive quantities of files and directories, and it's incredibly fault-tolerant. What's wrong with NTFS?

Re:But...I like NTFS!

[tomstdenis]

It's not leading .NET edge. See .NET MSFT wants to .NET drive home new .NET buzzwords that will .NET get customers to .NET recognize their .NET brandnames.

It's almost as if MSFT is trying to "innovate" again which is almost never a good thing.

Hopefully MSFT will be smart and allow users of the new LongBuildSlowReleaseHorn windows turn off this extra-probably-wastes-a-gig-or-two database.

Tom

Re:But...I like NTFS!

[RossyB]

The article explicity states that NTFS is not going anyway, as it works. Did you actually read the article?

Re:But...I like NTFS!

[Fly]

You're an _____. Read the article. WinFS is all about metadata, stored on top of NTFS.

Re:But...I like NTFS!

[InfoVore]

What's wrong with NTFS?

It doesn't force people into functionally unnecessary upgrades which keep MS profits high, that's what's wrong with it.

"Microsoft - We force you to upgrade because We Care (about our profits)".

Re:But...I like NTFS!

[penguin7of9]

NTFS has been an excellent FS for years! It's fast, it supports massive drives and massive quantities of files and directories, and it's incredibly fault-tolerant.

Yes, compared to (V)FAT, it's a great file system. But don't worry, Microsoft is trying to give you that old DOS feeling again by adding ample new opportunities for data loss and performance bottlenecks.

Re:But...I like NTFS!

[NineNine]

I'm a series of 5 underscores??

Re:But...I like NTFS!

[mbourgon]

WinFS is all about metadata, stored on top of NTFS.

Yeah, right now it is... as opposed to several times in the not-so-recent-past, where it was a replacement filesystem. It's possible the most recent explanation is the truth, but none of us will know until it's out - this MS guy may be wrong, he may be feeding disinformation, or they may have figured out that they can't build a DB-oriented FS, and have come up with this way to get something like it.

Re:But...I like NTFS!

[EngMedic]

one problem with NTFS i have is that it's read-only under linux, which is a pain... especially if, like me, you dual boot. or, also like me, do tech support and suddenly realize that knoppix is hamstrung by it's inability to write to a NTFS partition (and since all XP-pro and win2k installs are NTFS... this is a problem.)

Re:But...I like NTFS!

[BigRedFish]

What's wrong with NTFS?

The Linux/BSD crowd has already figured out how to read it, and are getting close to being able to write it. Pretty soon people might be able to dual-boot and have full r/w access to their legacy files from a non-Windows OS, without even having to back up all their data and reformat/repartition. That's what's wrong with NTFS. MSSQL to the rescue; those penguin-people and Lindows-guy will never figure out how to read this new filesystem. Well, until they do. But then it'll change again.

Why XML? It's much bigger than the old binary 'chunk' system, so the filesystem will grow huge. Selling hard disks is not the point, MS doesn't make those. The point is, with the criminalization of DVD burners and the 800MB CD limitation, to make it nigh on impossible for Joe User to make a full system backup so that he can migrate to another platform. A second hard disk (hello 'activation') wont do any good if it can only be formatted with the same incompatible new format. The idea is, by bloating the filesystem beyond the practical limits of common consumer storage media, and making it impossible to format a large hard disk in a cross-platform compatible way, only the hardest-core geeks would even think about switching away from Windows.

That's what the much-hated FAT-32 size limitations are about too - if it weren't for corporate users, they probably would have pulled FAT-32 already for this reason, or at least put in an artifical size limitation on reads as well as formats so we couldn't use FAT as a go-between anymore.

Never mind replacing your favorite apps, without a second machine (Windows tax paid again, ka-ching!), if you switch away from Windows how ya gonna get your data back? The only reliable way I can think of would be to network the old Windows box and the target *nix box together, run a black-hat crack script against the Windows box, and then download its contents via HTTP:80. Joe AOL won't be doing that (even if every website he visits probably is), never mind networking and Samba, assuming MS hasn't broken that too.

In summary:
void configureNewWindows(int version) {
kernel = NT4_kernel() + superfluousVersionCheck(version);
filesys = NTFS() + superfluousIncompatibilityByte();
filesize = backupMediaMax() + 1;
}

[Yes, I know that MS code should be in Hungarian obfus^D^D^D^D^Dnotation]

Re:2006?

[TheRaven64]

Or use the Be implementaion by this time five years ago.

And where is the inventor of BeFS now? Apple. I'm expecting so see some serious FS innovation in MacOS 10.4 (codenamed `pussycat').

The nicest thing I've seen using BeFS is a partial Jabber client that exposes each online person as a file, which can be viewed and messaged from the Tracker.

Oooooooh, yeah...

[Noryungi]

a filesystem which, accoring to Microsoft, will open up a whole new world of information availability. ... for virus writers, spammers and crackers, that is.

No, I am not trolling. I am not even being funny. If Microsoft cannot reasonably secure their OS, what makes you think they will be able to secure this new file system?

Think about the possibilities: no Admin password on this Windows 2020 box you just r00ted? No problem!! Just do the following SQL request and you'll find the password in plain text in this database table.

Oh, and while you are at it, just do this other SQL request to find all the documents marked "confidential" and this one to find the pr0n stashed away somewhere on the HDD.

It's going to be really ugly out there folks...

Re:Oooooooh, yeah...

[kabocox]

Oh, and while you are at it, just do this other SQL request to find all the documents marked "confidential" and this one to find the pr0n stashed away somewhere on the HDD.

Damn, I see an easy way of filling up those future 2 Tb HD's.. A PP Porn SQL Worm that searches for all videos and pictures marked as porn or with a porn related term and makes sure all this information is spread throughout the internet.

Would this worm be a good thing or a bad thing? I'd think the Regular Joe's would love having their porn collection increase automagically. Regular Joe's Wife might not like it though.

File now or file later

[TheConfusedOne]

Now, admittedly, this would also add on some responsibility to tag keywords to the files, and I've thought of ways of doing that as well (for example, applying keywords associated with a directory automatically to files placed in the directory).

Of course you'd have to do that scut work for any of these FS's to be useful. Now if you've gone to the effort of making the directory meta-data useful and explanatory then wouldn't just walking the directory tree accomplish the same goal while being less complex and more compatible?

Suppose you go and move your files (say the 'photo me sophie' now needs to go to 'photo me annoying-ex' folder :-D ) are you going to be redoing all of your indexes and everything to support that move operation? (Considering right now all you'd have to do is rename and possible move the folder [maybe to the 'trash'] to get the job done.)

DB backed file systems are only really necessary when you're dealing with document management and with documents being created by other people, not yourself. The one thing you learn with DM implementations is that those DB's containing "meta-data" are always stuffed with the data that is least annoying (not most helpful) to everyone using the system. Context is quite important when you're viewing data and what you may consider important for your search could be completely useless for the next person.

Re:File now or file later

[FreshFunk510]

"Now if you've gone to the effort of making the directory meta-data useful and explanatory then wouldn't just walking the directory tree accomplish the same goal while being less complex and more compatible?" Maybe I didnt' read the article closely enough but I had the impression that the meta-data would be created by itself. As far as all my experiences have been, meta-data is usually something automated.

Anyway if Microsoft can enable a google type search on my personal hard drive I'd be really impressed. That's what it sounds like they are going for. And I do find this useful because:

1) Not every home user is a geek who micromanages the organization of their filesystem.

2) Hard drives are still growing. Even know if I only navigate through parts of if leaving many of my mp3s untouched. If I could access all my Cure mp3s simply by seraching for "Cure" and getting results back in
3) Yes, if you organized everything yourself maybe it woudl be easier (i doubt it ..not if you had a good search and a whole bunch of files) but the question, then, is usability. If I don't have ot organize anythign but search can return a good result set (a la Google) of course it would be a valuable feature.

Re:File now or file later

[Keeper]

Suppose you go and move your files (say the 'photo me sophie' now needs to go to 'photo me annoying-ex' folder :-D ) are you going to be redoing all of your indexes and everything to support that move operation? (Considering right now all you'd have to do is rename and possible move the folder [maybe to the 'trash'] to get the job done.)

Updating that information would just be a sql update statement. How hard the UI to use it is a different story all together.

Re:File now or file later

[cpopin]

Is that how you get rid of annoying ex-girlfriends? I want a file system like that? Does it work for ex-wives?

Odd statement

[mcc]

I mean, yes, it certainly is nice that Microsoft is managing to implement this specific OS feature before Apple or the Linux community does, and they should get a gold star for Super Effort, but still.

If this were the "holy grail", wouldn't people have cared a lot more about it when BeOS did it ten years ago?

I find it amusing Microsoft has now for years been harping on this one SQL-drive feature in an OS that keeps getting pushed back and back. I guess it's because it's the only thing Microsoft has done in a long time that actually seems like innovation, since it's never been done before in an OS that most people have heard of.

Can it get any more open?

[phallstrom]

"...all into a filesystem which, accoring to Microsoft, will open up a whole new world of information availability."

Seems like with all the worms and viruses out there that Microsoft has already achieved this goal :)

Google for the desktop?

[wagsworld]

So let's see we are now going to have yet another processor consuming waste of resources that helps maybe 10% of the population figure out how to use the damn search function. I am sure this will not affect performance, security or stability because SQL Server is the premier Database engine in the world. * Cough **Puke* If, and it's a really big IF, MS can manage to get this to work in more than their test environment who really wants it? I don't hear people screaming about not being able to find files or data on their own machines.(On the internet is a whole different story) I hear people almost every day screaming about the damn number of patches they had to download when they reinstalled. Then there is the multi annual DDOS Virus issues. This more of the wonderful MS misdirection..... Don't look over here at the real problems we will yet again leave unaddressed here is this new feature you just have to have. You won't be able to live with out it. And like the crack dealers they are they will leave us all wanting it but never invest the time to make it work right. We will get to add it to the list of useless apps in Windows that everyone should turn off. This is yet another Micro$oft BOB Application. Does anyone remember MS BOB. That wonderful little App that didn't last a year that was supposed to make computers even easier to use. Anyone who did use it removed it in less than 6 months because they realized it was causing a whole host of problems and never did help them to use the computer. It merely masked all the really difficult to use things like control panels and the Start menu. MS needs to get over themselves and fix what needs to be fixed. I don't care about new features right now I want a reliable stable OS. I would use Linux or Mac OS X full time if I didn't have a job in IT. This is just gonna cause still more pain. Just another reason to start recommending that Companies at least switch to Mac OS X if they can't stomach Linux.

Re:Google for the desktop?

[Knights who say 'INT]

Why doesn't Apple port OSX to PC's - or at least, to Intel-based PPC's? Any serious technological issues?

Boy, it'd make quite a dent. I can see myself shelling out cash for an OS for the first time in my life.

Only cool when used correctly

[jawtheshark]

Now, admittedly, this would also add on some responsibility to tag keywords to the files,

And this is the point where it will fail. Look, the concept is nifty, that is true. However common users rarely bother to give a document a good name, so why would you think that they are going to fill in the metadata? (Which you can do by the way in Office documents)
Now imagine this: we are in 2006, you have your digital camera and come home from a long vacation. You want to upload the pics to your machine runninng Windows Longhorn with WinFS. Ah! Right now you get something like 00001.jpg, 00002.jpg etc. How will the camera know what metadata to put for the pictures? It won't... The only way to *enforce* it is to show each and every single picture and prompt for relevant metadata, which of course no sane user is going to fill in. (Imagine doing it for a couple of hundred pictures) It is that simple. Right now you would just create a folder "vacation to Florida May 2003", dump in all the files and be happy.
No, you won't find the files of you and Sophie the classic way.... but unless the metadata entry is correctly entered, you won't with WinFS either.

Re:Only cool when used correctly

[merdark]

Umm. You'll still be able to dump them all under a header with WinFS as well. And cameras DO embed a lot of metadata into the photos. This data includes the camera model, focus length, and other such setting. Plus, later you could go and comment all your pictures. I do this for some of my vacations when I put them on a web page. It's not unreasonable.

Re:Only cool when used correctly

[EddWo]

Well the jpeg exif format that most camaras use has a certain amount of information. Camera manufacturer, model, exposure, flash mode etc. If your camera has a clock it will store the date and time the picture was taken. What I personally would like to see is the camera also storing GPS coordinates, but that might be rather overengineering, perhaps only phone cameras will have that kind of feature.

Part of the WinFS project will be to create lots of import translators that detect when files are being added to a WinFS drive and read out the standard metadata.

That level doesn't include who is in the photo but it does provide a lot of information for locating it. I expect you could easily apply a tag to a whole set of files at once such as "Holiday Snaps".

With a database and metadata standard in place theres no reason why third parties couldn't implement other cataloging features like facial recognition into the system. Just show it some pictures of your family and friends and it will pick them out in your other photos.

One of the pushes from MS is for third party developers in include a metadata standard with all the files that they save. A lot of the metadata needed can be provided by the program itself at the time the file is saved.
I expect the programs will start to ask a few questions, "What projects is this file related to?", "Which contacts does it involve?", In this way much of the metadata is created as part of the natural workflow and it doesn't feel like an massive exercise in tagging everything.

The Blue Screen of Corruption

[Camel Pilot]

Let me the first to turn the obvious phrase of 2005.

BSC like BSD has a nice ring to it.

A better question...

[CooCooCaChoo]

I know I'll probably be called a troll but why on gods green earth are you running Windows on a server? sure, I can understand why one would need to run a Windows desktop, but a server?

The best way to remove the missery and heart ache of Windows is to simply buy that $100 per employee pack from SUN and be done with it. Everything one needs to get a server working; application server, Instant messenging server, directory server, etc etc. oh, and most importantly, Solaris on x86. A dream when it comes to a server OS.

Sure, Solaris sucks huge chocolate salty balls as a workstation but as a server, as some youngsters would say today, "it r001z!"

Re:A better question...

[CAIMLAS]

I've got news for you, NTFS doesn't handle well on a desktop, either. :) At least not as a stand-alone workstation. If you start to get any significant amount of data on a volume, performance goes to hell. (even 30G being significant in this case)

Re:Price of admission is a full disk compr[o]mise

[Haeleth]

> MS users do not require such complex systems...

WTF? Here I was thinking that Microsoft Windows was widely used in industry, government, and academia, for tasks for which such complex systems will be a great benefit indeed. (Whether said users might be better served by a Unix-like system is another question entirely; the relevant fact is that they use Windows.)

WinFS will also benefit home users of the sort whose standard method of locating a file on their hard disk is Start/Search, and who will not recognise it as a sophisticated system at all - all they will notice is that Longhorn lived up to the advertising claims that it would make searching for files faster and easier.

Perhaps they a working text search first

[gonkem]

My god!

How utterly crap is the text search in Windows XP. It doesn't search everything ever, it's an acknowledged feature in the KB, with a workaround.

The workaround lets you search additional file types, but not everything. Utter braindeadedness.

It's not hard... well you'd think it wasn't... but apparently it is...

Break Samba

[Czmyt]

I see some benefit in this new filesystem that Microsoft is working on, such as the ability to distribute files across servers using the same technology that lets you distribute database records now, but I think the real reason they want this filesystem is to break compatibility with Samba in a major way. This does not sound like it is going to be something that will be easy to duplicate with Open Source software.

Who's the next Netscape?

[tjamme]

Call me paranoid, but despite first thinking that it was a great idea and that for once Microsoft was doing somehing original, I'm now thinking they want to do a Netscape on Oracle and MySQL.

If their OS integrates database functionality, that looks to me like a new back-door entry into the database market, once again confusing OS and apps.
The best way for the Open source / Free software community to react is, I think, to see convergent efforts to maybe integrate, or provide hooks, for something like MySQL (or others, PostGres?) to work with Linux in the same way WinFS proposes to.

--Teebo

Lack of Ideas

[awol]

There are many things for which MS can be blamed, but one of the more powerful criticisms is that they have never done anything "truly wonderful" with all their power.

I think this File System idea is just another example. Everyone talks about a "new" file system as if adding a relational database (of some kind) or meta data for the existing concept of file is somehow revolutionary.

Why not try more radical ideas. We have a document style that is based on a generic "template" some of the document is the same but there are thousands of instances of the document that have different content. Why not when I save instance X does it not just split the document into "same as all other instances" and "different" and store these fragments either as formal records or as references to the original instance. The complexity of "how" to do this would be enormous, but the fragementation could be handled even better to allow content creators to "farm" content from their existing data without having to know about the formal nature of the whole entity from which the data comes.

Or a file system that does not use the concept of file as such but rather a whole CVS like "history" so that the entire change history of the entity can be managed, monitored and retrieved from any point in time. Then when one moves the entity one could move a sna[pshot or the whole thing.

Or a memory based file system where there is no distinction between physical memory and hard disk and the OS treats all data access as an access to memory and cachjes out to disk invisibly underneath.

Three simple ideas that are more interesting (in my view) than just adding an RDBS layer on top of the existing hierarchical FS concept to help manage meta data more easily. You still need to create the meta data, more pain to try and keep it up to date.

Re:Lack of Ideas

[WuphonsReach]

VMS has auto-revisioning where it would store (complete) copies of older revisions of the file every time you saved. So you'd end up with a directory filled with:

PROGRAM.C:1
PROGRAM.C:2
PROGRAM.C:3

I don't know of any desktop file systems today that do that (maybe NTFS?). (That was back around 1990, when a 20Mb HD was huge.) It's definitely something that I wish would be built into a file system today (especially now that we have oodles of HD space).

Re:Lack of Ideas

[quantum bit]

Or a memory based file system where there is no distinction between physical memory and hard disk and the OS treats all data access as an access to memory and cachjes out to disk invisibly underneath.

Most modern OS's already do this with the VM/buffer cache.

Data labelling and ACID complience seem quite good

[pfafrich]

Well everyone else seems to think this is a bad thing but I can seem some postitive aspects.

Using SQL might give ACID complience which could protect data. No more lost/corrupted files when Word crashes.

Data labeling through xml seems like to could offer some advantage. I've often wanted to add a brief note to my files explaining what the file is. Say for a jpeg image have a text description of the image. May be I could add some tags or Key Words to the meta information about the file. This would help me search.

Whilst a tree structure for files/directories is probably the easiest way to organise things it does have limitations. Symbolic links don't always do the job. Find is currently way too slow maybe something with the simplicity of google's search box is the easiest way to get to your data?

Maybe the fs will be able to cope with different namespaces providing a simple interface to all different resources.

It seems like MS are trying to do something inovative and this for one should be aplauded.

The holy grail

[salesgeek]

I've already got one and it's very nice!

No need to wait!

[amightywind]

In particular what I'm looking for (and maybe other people are too), is a file system where it's easy to find files.

It already is on any LiGNUx file system!

% find . -name "*me*sophie*.img" -print

What could be better?

Re:No need to wait!

[Demandred]

I hate to be antagonistic, but there are several problems with the method you propose. That model requires that all the attributes that one would wish to search for be embedded in the filename when it really should be meta-data for the document. Also, your queries would become more obtrusive since you would also have to do:

% find . -name "*sophie*me*.img"

if you are searching for more than 2 attributes, this becomes a pain. Furthermore, the mechanism you describe would also retrived a document titled "sophie_in_mexico.img". So, simple regular expression pattern matching does not suffice for that type of search.

Plain Old Text - Easy to Read, No Wonders its #1

[Rupertx]

DEVELOPER: I seek the Grail! I have seen it, here in this operating system!
BALMER: No! Oh, no! Bad, bad Bill!
DEVELOPER: What is it?
BALMER: Oh, wicked, bad, naughty Bill! He told everyone that we have a super-cool operating system which, I just remembered, is grail-shaped. It's not the first time we've had this problem.

Oh Microsoft...

[base_chakra]

We start with an inferior file system, and then mmmmmmmmm... we pour on the overhead. First, a creamy layer of laughable SQL database implementation... then we top it all off with a questionable application of a steaming hot technology. Ooh, baby it's like a huge performance boost in my mouth.

Eat it fast before the indices self-corrupt and you lose all your data relationships.

Re:Oh Microsoft...

[gregarican]

No doubt. I remember back in the day when file allocation tables or boot records got hosed and hard drive contents were lost. You could "fdisk /mbr" or scandisk or else grab your ankles.

Seriously though, I don't see MSSQL having data integrity issues as much as security vulnerabilities. A new worm can hose up your file system as well then.

Holy Grail, Batman!

[freeze128]

It's the Holy Grail for developers.... Maybe the developers should learn to organize their files better. This is just going to lead to sloppy programming proctices and lazy programmers. "Why keep the files in directories that make sense when all we have to do is search for them?"

Well its still there you cretlin

[EdMack]

Now you just have an easier way to search it.

Information Accessibility revolution

[Junior J. Junior III]

was pre-empted by the Windows Security Development Task Force. Now that they've handled things, everyone has access to everyone else's data.

Samba ?

[bl8n8r]

Can the SAMBA team expect to have a white paper available to them documenting the standards to implement for this new filesystem? Oh, wait.. that would most likely defeat the primary purpose of WinFS.

Typical /.

[delus10n0]

I like how 99% of the replies to this article are negative or uninformed. We're really delving into the land of the SlashDot, aren't we?

"It will be horribly slow!"
"It might be insecure!"
"NTFS is all we need."
"LiNuX R000lZ!"
"MS SQL is stupid, mySQL is better!"
"XML is dumb, no one uses it."
"In Microsoft WinFS, the files database you!"
"Imagine a beowulf cluster of these.."

Re:Typical /.

[delus10n0]

Clearly you are not a real beta tester, otherwise you'd understand what "beta" means.

And I'm glad you're an expert on filesystems, too. Perhaps you can help them write WinFS!

While its easy to poke fun...

[222]

An SQL based file system is long overdue on the desktop, and as much as it may hurt to say it, Microsoft is actually somewhat on the ball here. I'll probably use longhorn when it comes out, and ill probably be impressed with all of the new bells and whistles.
It really doesnt look well for open source if we only see features in linux months or years after theyve been in windows... and it really enforces the idea that open source software relies on the copying of ideas, not innovation.
Longhorn isnt due out for a few years, so why not prioritize features like these now? It sure would be nice to have an sql based fs and an interface rivaling OSX *before* longhorn is released....

Not neccessarily bad

[claes]

Well, this does not have to mean what most people here seem to think.

What the article says is: WinFS will use the "querying capabilities" of SQL Server and the "data labeling capabilities" of XML. Now, put this into context. It does not neccessarily mean that everything will be stored as XML. I imagine it could mean something like this.

Think of a directory as an SQL table, where the file name for each file is the primary key. Other metadata for each file, such as size, timestamp, priviligies can be thought of as other table columns. With SQL you could imagine doing queries against this directory (think table). For example

directory filename from /home/joe where filesize > 1 MB and mtime 1 day ago

would list all filenames for recent, larger files.

Now, if you want to make more detailed queries, files could have specific metadata added to them. This metadata could be described with xml. For example, the available metadata for mp3 files could be described with an xml file that describes what metadata you can search for files of type "mp3". In a way, you extend number of columns in the table. XML can be used describe for the querying engine what kind of other columns there are, for this kind of file. It can be though of as a bridge between the quering engine and the underlying storage format (in this case, mp3 id tags). Think of it as a plugin description format (perhaps similar to how Hans Reiser talks about plugins for his future versions of ReiserFS). It does not need to be attached to every file, or be the storage format for every file.

This is just speculation, but they are not stupid. They will not store everything in xml just to be buzzword compliant

Re:Not neccessarily bad

[peter]

> directory filename from /home/joe where filesize > 1 MB and mtime 1 day ago
>
> would list all filenames for recent, larger files.

And you think this would be faster than what we have now?
find /home/joe -type d -size +1024k -mtime -1
(and you can combine find's arguments with and or or logic, like with SQL. Obviously you don't have the same power as SQL with just find by itself. Traversing directories doesn't take that long, esp. if you've already done it recently so everything's in the cache.)

doom scenario

[stock]

If Wintel succeed to ban Linux and Open Source, Longhorn will be swiftly introduced, Palladium and DRM are implemented in hardware, and all people who are cought running Linux will get to state federal prison, as cheap labour.

Next data recovery will become a enterprise class only service, which will cost fortunes. People might start loose data on their PC's as frequently as today BSOD screens popup. So next Wintel strongly suggests to put your valuable data on their subscription storage sites.

Robert

Re:WinFS == SLOW

[gregarican]

They should have gone with a journalling filesystem like many of us wanted.

Eh? Who wanted? Not me. For what it is I don't have many problems with the NT/2000 file system structure in general. I know VFAT, FAT16, FAT32, all had their integrity issues and other problems but NTFS has seemed to be adequate for years now. If it's not broken (if someone can point out why it's broken please do so) why fix it?

NTFS is NOT dead, read the article!

[Kevinb]

Read the summary, even. I see a lot of people here complaining about NTFS going away -- it's not. WinFS is being built on top of NTFS, such that the underlying filesystem can still be reached.

Think about it: It doesn't make much sense to move (for example) the kernel and other low-level files into WinFS because (a) it raises bootstrapping issues (in order to load the kernel, you'd have to bring up the database engine, but the database engine depends on the kernel) and (b) they don't contain much, if any, useful metadata to search on. So logically, the OS will still be living in -- and accessible via -- the traditional filesystem.

That's not the point

[RealisticWeb.com]

Of course there are major technological issues, such as not only completely re-writing their kernel, but also completely re-writing the entire Carbon framework. But besides that, the major issue is that Apple is primarily a hardware company and always have been. They make some of the best software on the planet and the whole purpose of it is to drive the sale of more hardware. If they let you install OS X on commodity DIY x86 parts, they don't get a hardware sale, and that would eviscerate their business model.

Filename conventions

[GodSpiral]

What can WinFS accomplish that couldn't simply be done through filename conventions.

For example:
My Word Doc.doc
My Word Doc.doc.xmlannotations
My Word Doc.doc.sqlindexedtextversion
My Word Doc.doc.xmlenterpriserelevancy

together with a bunch of services that monitor, and assist in generating, querying, and interpreting the data.

MS may be providing tools to assist in all this, but isn't this a very simple idea that can be done/copied with any existing OS, with little effort?

Isn't this just slocate with more metadata?

[electroniceric]

I don't get it. How is WinFS any different from indexing files and storing the metadata in a database, a la slocate but with more metadata?

As I see it, if an app that doesn't know about WinFS, say a legacy app (or open up its data storage to WinFS) goes to store data in WinFS, it will ignore most of the metadata layer, and store its data as binary, whether that's as a BLOB in a DB or a file pointed to by a fs tree seems immaterial. This says to me that the main problem is that apps don't TELL each other how to share data (i.e., open data formats), and I don't see how WinFS addresses this. Am I missing something here?

I really don't see how the XML fits in either - if we want an XML filetype, why not just make an XML filetype, and then index afterwards?

Everyone's looking at it the wrong way...

[psykoz]

Alright, just to get it out the way I am not a MS supporter, however I feel everyone is looking at this the wrong way. Particularly what Microsoft's intentions here are clearly to speed up network browsing. What they intend on doing is the same way lots of metaframe server farms work (citrix) and other server-server communication. What they will end up doing is when you do a search for a file from your computer on the network for x server your filesystem will send a SQL query (via XML) to the network server and then the server will do a search for those files and then send the list of their locations back to you immediately (via XML). Another client-server method that they're trying to pave the road for. However yes it is more moving parts, so more room for problems, but if they do get it working I could see it being nice to an extent being able to communicate the file system via query.

Inspiring!

[MarcQuadra]

Your post has inspired me... ...to have my eyeballs removed. :-)

Oh Boy!

[ronmon]

WinModems, WinPrinters and now WinFS.

The first two were such huge innovations I just can't wait for the next one.

Re:2006?

[nFriedly]

not in my copy ;) btw - i DID go out and pay for a legal copy of xp before i 'patched' the activation problem i just install a lot of new hardware and got pissed off after i was asked to activate my xp twice in the same day

Somebody Left the Grail Beacon On...

[HopeOS]

How long has Microsoft been working on this now? It's not going to happen. WinFS is fighting against entropy and human nature. It's going to lose.

The moment you enter meta-data it becomes stale. Unless the meta-data is derived from the data itself, it cannot and will not stay synchronized without manual intervention or diligent application coding.

If the application is responsible for maintaining the tags, only standard meta-data fields can be used or applications will not interoperate. Can we expect the equivalent of ID3 tags for all user data? Of 100 potential industry-standard tags, how many will you present to the user to fill out before writing a file to the OS?

Will you be able to search on vendor-specific tags? What if there are over 2000 unique tags on the system? Was that "creation-date" or "creation-time"? If I specify both will I get the intersection or the union of the data? This is starting to sound a lot more like report writing than a google search. SQL queries are designed to operate on normalized data of a known format. That's not going to happen without OS level enforcement which brings us to the next problem.

If the operating system is responsible, somebody has to write code to parse these files and extract the relevant tags. Will that code be signed? What user context will it run in? What about proprietary file formats? Who decides what code does the parsing? More importantly, what happens if the data is invalid or worse, deliberately malformed. IE was compromised this past year by a flaw in the MIDI library. Can we expect to be rooted by simply indexing a rogue file?

Then there is the issue of having multiple copies of the same file with changes too subtle for either the application or the operating system to detect. Presently, we distinguish these files by name, but we also frequently distinguish by location in the filesystem hierarchy. For instance, all images under directories red, green, and blue may have slightly different hues. How shall we tag these? And if they are generated by a batch script?

The two most important attributes of a file are the user given name and the position within the filesystem. In fact, those are the only two details that general applications today will even ask for before saving. Why? Because it's hard enough getting people to choose a better name than "letter.doc".

In closing, I don't see how any amount of indexing and tagging will justify the overhead on the filesystem, the burden on developers, or the potential security risks that this scheme entails. MS will lose a lot of momentum working on WinFS.

There are better approaches to the "lost file" problem.


-Hope

It's about VIEWS on data

[Otis_INF]

"But I like NTFS!"... and other remarks... I don't get them. Sure you probably like the 'File' view on data on your harddisk. That's the only view NTFS can deliver you at the moment.

However that's inefficient. The reason for that is that when you have a word document with an embedded table, and you want to read that table, you have to open the document. What if you could see the document FILE as a view on the data that make the file: text and that table. Then you can also create another view on the same data: just the table. Or other text and the same table.

THAT is something NTFS or another FS will not bring you. With this FS it can. That's the power and the holy grail the VP is talking about. After all, RDBMS-es started out as a lousy file system too, but today data isn't stored as files on disk, but much more efficient to make working with the data more efficient.

It could just be me ....

[jacoby]

But it sounds like a Windows Explorer replacement rather than a significant change. Of course, I got sick of the buzzwords and lack of specifics. Bolting a rdb on the side of a file system? Be did that and backed off, didn't they?

This is an ad

[511pf]

This is a freaking advertisement for a vaporware product that may or may not be out in a few years. Microsoft should buy ad space on CNet...oh wait...

RTFA

[wishiwascool]

The quote was mentioned out of context
Here is the real quote:

"The desire has been around forever. It's almost like the Holy Grail of data storage," said Michael Cherry, an analyst at market researcher Directions on Microsoft.

The MS VP did not say that WinFS was the "holy grail". Someone else described "the vision of a single storage system that would break down barriers between applications and serve up stored information quickly and accurately" as being a "holy grail".

Everyone is so quick to bash MS...

Stu
(I program on both Windows and *nix OSes)

Wow, Now That's Innovation

[Bravo_Two_Zero]

So Microsoft is going to make a filesystem that does essentially the same thing as Oracle Files? Will the innovation ever cease? But, I'm sure all the Windork developers where I work will hail it as a great advnacement, just like VB Script!

Re:I'll Buy It

[gregarican]

It's called MSDE (Emmmm Essss Deeee Eeeee) and it's already free. Kthanx.

Re:Thoughts on XML - lucid parent post

[jemele]

Disclaimer: This post may be off topic

My roommate and I have discussed this point for quite some time. He had an interesting point relevant to the prevalance of xml everywhere, including rpc mechanisms.

Proponents argue that it is human readable AND an industry standard that allows inclusion of metadata (other data) with data.

Nothing to argue here.

My roommate and I eventually concluded that for mechanisms like rpc, where machines are the only participants in a dialogue, of what value is human readability?

Metadata is useful; since languages exist that are capable of implementing the semantics as xml (xslt), this is good (but lisp did it first).

Providing tools to do all of the above and while keeping it effecient strikes me as a more well-reasoned and prudent approach to computing.

>> does anyone else think using XML in a filesystem is a horrible way to go?

XML in the filesystem: Why write what you will never read?

tchuss.

speaking of filesystems...

[the_greywolf]

does anyone know if there's an interface i can use to write a ext3 driver? :)

Re:speaking of filesystems...

[EddWo]

Do you mean the Installable File System Development Kit?

Performance and Volume

[chiph]

For us at work, the hot issues are:
- The ability to handle millions of little 2k files.
- Performance

The two are somewhat opposite goals, but maybe the new WinFS will be better than the combination of NTFS and Microsoft Distributed File System.

Chip H.

Re:Performance and Volume

[gregarican]

I know personally that I had to modify settings for Windows 2000 workstations to browse SMB/NTFS network filesystem resources better. Can't wait for patches and service packs for a new multitude of performance and security problems that WinFS will create. Details on my issues with Win2K are below:

Slow NTFS Browsing For Windows 2000 Clients (Q265396)

SYMPTOMS

When you select a file on a network share that uses an NTFS file system partition, extra Server Message Block (SMB) packets are sent on the wire. This behavior causes overall slow network performance.

The same issue occurs if you browse the network, and you position the mouse cursor over one of the files; however, this issue does not occur if you use a Microsoft Windows NT 4.0-based client computer.

CAUSE

This issue occurs because Microsoft Windows Explorer tries to open NTFS streams on the volume to get extended information about the file, for example, title, subject, comments, and source author. All of this information for the file is stored on the volume.

BeOS...

[Space]

Hmm, a relational database overlaying a file system...
Where have I heard of this before...

^MOD^ ^PARENT^ ^UP^!^

[BOOTSTRAPS]

^MOD^ ^PARENT^ ^UP^!^

data format lock out?

[Bizzarobot]

WinFS will address concerns in how developers create applications, Muglia said. Whereas programmers had to deal with a separate data format for each type of application, WinFS provides developers with a new storage option designed to bridge all application types.

Does this sound like a system-level implementation of Office file types? As application developers use WinFS as the storage type for their data, MS is then in control of who and what other applications and operating systems get to use it. If another platform develops a "client" to WinFS files, MS can tweak the next release to disallow anything but Windows to read the data. Like .doc, .xls, .ppt, and IM protocols have been so famous for... no?

Re:Thoughts on XML - lucid parent post

[I8TheWorm]

You, sir, should be modded up! :)

Still haven't fixed the old file system

[edxwelch]

I upgraded from WinNT to WinXP and now the file system crawls. Where before to open a folder was instantanous, now it takes several seconds. I tried every tweak and trick that I could find on the internet, but none of them make any difference. I think it's simply Windows XP has a performace problem with SCSI drives.
So, now we get a new even slower file system, probabaly switched on by default.

Re:Still haven't fixed the old file system

[gregarican]

See my post regarding a similar issue to this. My issues were with Windoze 2000 boxes but Windoze XP might be in the same boat. Dog slow file browsing for sure.

Re:Still haven't fixed the old file system

[WuphonsReach]

How much memory do you have installed? WinNT needed around 64Mb to run happy, Win2000 128-256Mb, WinXP wants 256Mb-512Mb.

You know, I like this idea....

[Anonymous Coward]

But I like this one better

There is a real chance here for a revolutionary change in OS design.

But hey, why innovate, when you can cripple new ideas with 'backwards compliance' and make big piles of cash?

Gnome-Storage??

[NomadResident]

Wow!! Sounds like M$ is reaping the rewards of the hard work being done at Gnome on the Storage project (Supposedly going to be released with Gnome 2.6). Typical "The best Ideas are stolen" philophisy. Hopefully Gnome beats M$ to the punch!!

games?

[happyfrogcow]

How would gamers like this? gamers typically want superfast access to data on the disc, and don't care how the data relates to other data that might be on the disc, no? the only thing maybe good i can see about this is that of cached queries, but would that help any? enlighten me.

Re:games?

[EddWo]

Thats why its NTFS underneath. A game knows exactly which files it needs to load and where tehy are stored so it will just use standard file system calls.
A productivity suite will show a dialog box with a query interface for the user to seach for the files. It will get from the database the actual path to where the data is stored an load it in the normal fashion. Its more of a shell extension than a new file system its the best of both worlds.
You don't need system files and program files to be indexed in a database, most people don't spend time searching for them, but user created files and digital content will be indexed for better searching.

Re:Worst? You don't know how bad things can get!

[cayenne8]

"I take it you don't consider MS Access a database system."

Nope...I don't and it should be banned from the desktop...

It's like Googling your hard drive

[kylef]

This seems classic MS-predictive-FUD, where people hold their breath for the Next Release, which is a 1.0 version that sucks.

While I don't claim to know what version 1.0 of WinFS will be like, I don't think the idea of executing a system-level "metadata" search is a bad idea at all.

A filesystem is basically a clumsy database, with the fields being rather arbitrary "directories" and "filenames." The concept of a directory is not intuitive for computer illiterates at all. Allowing users to seach for their data, without worrying about the concept of where it physically (or even logically) resides, is a cool idea.

Lots of regular computer users don't understand the concept of directories and a directory hierarchy. But just about EVERYONE understands the idea of a Google search. I think that is the underlying principle behind this push.

Re:It's like Googling your hard drive

[spitefulcrow]

But I don't want google for my HDD! The metadata necessary would waste space and other resources on my system to implement something I'd never use because a) I understand the *nix hierarchical fs structure and b) if I don't remember where something is, I'll use ls with grep, or just locate. Even more of a reason for me to stick with Linux. MS is just adding features that not everybody wants. By serving the interests of the nearly computer-illiterate users, they bloat their system and make people who know what the hell they're doing like their OS even less.

Re:It's like Googling your hard drive

[kylef]

LOL. Like Microsoft cares what a typical Linux user wants. That's funny.

In fact, I bet they have people working in shifts working on your request right now. I bet that's their next big marketing blitz: Longhorn, now with arcane Unix command-line tools! Can you imagine the sales boost that will give them? An easy 0.001% increase in market share, right there!

BTW: using dir with findstr will work on a Longhorn console just like it does on 2k and XP. And whaddya bet that WinFS itself is a service that can be disabled if you don't want it? Hmmm... seems like that would pretty much nullify your criticism.

Sounds suspiciously like...

[flaminghyundai]

*nix, where "everything is a file."

This nothing more than BeOS filesystem

[DrMorpheus]

The filesystem for BeOS had all the capabilities that the article was talking about AND

• It was fast AND
• It had journaling AND
• It shared data between applications AND
• It allowed for sophisticated querying AND
• You could pull the plug on the machine and it would reboot quickly (typically less than ten seconds) with no data lost.

So once more Windows stealing technology and probably getting credit for "doing it first". Ugh!

Re:This nothing more than BeOS filesystem

[danheskett]

You are melodramatic.

First, BeFS is neato and all, but its not what MS is talking about. Thats point one.

Second, innovation is the process of bringing a technology or improvement to the masses. Lots of inventions end up in the bit bucket, not the market because the inventors aren't good innovators. It looks clear that MS *is* going to bring WinFS to the market. How many users did Be ever have? I was one, I had one friend and then of course. So at least two or three. But not many more. I loved Be, but lets be clear it was an infant.

Third, even if it were a direct re-implementation of BeFS, this wouldn't be "stealing technology". That would be like MS taking ext3 and compiling it for Windows and selling it as WinFS. "Stealing" is mis-appropriating goods and sometimes ideas. The concepts behind BeFS are well known and been done before. As an academic matter, BeFS is a good implementation of well established principles. Can't remember his name, but the author of BeFS actually worked from a book called approximately "Practical Filesystem Design".

Bottom line is that you sit down and shut up. You clearly dont know whats going on here. What MS is talking about is putting a whole new level of sophistication in data storage on the desktop of 100 million users. It's a big deal, and definately newsworthy.

Re:This nothing more than BeOS filesystem

[Talez]

As an academic matter, BeFS is a good implementation of well established principles. Can't remember his name, but the author of BeFS actually worked from a book called approximately "Practical Filesystem Design".

Close but no cigar. Dominic Giampaolo wrote a book called "Practical Filesystem Design".

This sounds similar to the AS400 Filesystem

[Coins]

The AS400 filesystem is basically a database. Files are just a certain object type, programs are a different object type. It sounds to me like MS wants to build WinFS on top of MSSQL the way that IBM built the AS400 on top of DB2

Re:This sounds similar to the AS400 Filesystem

[SharpNose]

I was having similar thoughts, but I was thinking about Files-11 under VAX/VMS. I worked in VMS for a long time before I got into Linux, but it wasn't until I tried to do some Linux/VMS interoperability and read the docs associated with the Linux DECnet utilities that I appreciated how sophisticated VMS was in that regard. Files-11 had a whole metadata and stored-attributes thing going on, which I assumed was heavily exploited by DEC's database products like DBMS, RDB, and Datatrieve.

Can someone who knows or at least remembers more VMS than me refute or elaborate?

Debugging

[Simon]

My roommate and I eventually concluded that for mechanisms like rpc, where machines are the only participants in a dialogue, of what value is human readability?

Ease of implementation and ease of debugging for the programmer. That is the value of a human readable protocol. All other things being equal, a human readable protocol will be easier (read: $$$ cheaper) than a non-readable one.

When tying together databases, server, RPC etc etc in a business environment, the thing you are trying to connect to, or work with will never be properly documented and any documentation that does exist will be wrong. You will debug. You will read the protocol. This is law.

--
Simon

Re:Debugging

[jemele]

quote

Ease of implementation and ease of debugging for the programmer. That is the value of a human readable protocol. All other things being equal, a human readable protocol will be easier (read: $$$ cheaper) than a non-readable one.

end quote
(plain old text doesn't allow tags ... tsk tsk)

This too, is something my roommate and I discussed. It is important to debug the protocol, when things go wrong.

However, in debugging, one must create tools. Event if these tools are as simple as: ::OutputDebugStringW(wsz_request);

nevertheless a tool has been created.
To take a step further - most people debug using debuggers. I will not start a holy war on the merits of gdb of vc but a tool must exist.

Computers were never designed for humans to directly interact with on their own - the tool requires a tool.

What is the difference between the aforementioned tools and another tool that takes a protocol byte stream and renders it human readable?

One such tool that immediately comes to mind is an assembler, in reverse.

patiently awaiting your rebuttal.
joshua

Re:Debugging

[Simon]

However, in debugging, one must create tools. Event if these tools are as simple as: ::OutputDebugStringW(wsz_request);

The difference between a readable and a non-readable protocol is that you can just log or snoop on the protocol stream directly with printf() etc and not have to write anything to decode it. Also your readable protocol is usually some form of text, so you can also immediately use any text handling tool at your disposal. Creating debugging routines costs extra time/money.

nevertheless a tool has been created.

You can look at it like that, but the point is that the two tools have different price tags.

To take a step further - most people debug using debuggers. I will not start a holy war on the merits of gdb of vc but a tool must exist.

In a live production situation you often don't have a debugger. The best you can do is printf() or log what's going on. Hell, sometimes you don't even have shell access to the machine running the application.

What is the difference between the aforementioned tools and another tool that takes a protocol byte stream and renders it human readable?

time/$$$/effort. Measure it however you like.

--
Simon

This is NOT the "holy grail"

[spitzak]

I will describe the "holy grail":

1. There is ONE call to get any piece of data on the system. This takes a "name" which is a block of bytes and a length. No bytes are illegal in that block. This means you get the name of files, their size, the protection attributes, the data, the "resource fork" and everything with the same call. There are NO other calls that return information.

2. The information returned is a block of bytes and a length. Notice that this is exactly the same format that names are. You may have to use "read" and "seek" calls to look at this block of data, but you know it is a block of data. The block of data can range from 0 bytes in length to any number of terabytes, short files (less than 100 bytes) would be stored extremely efficiently.

3. When a filename identifies a block of data, adding a special divider character (ideally nul, but it will probably have to be "/") to the end and then more characters, will identify sub-parts or attributes of this block of data. This can be done by providing the block of data to a user-level program that further parses it (imagine a tar file or compressed), or a user-level program that even communicates with other systems (imagine a filename that indicates a document over http). Although these programs can interpret the text after the slash in any way they want, calling programs will likely assume that adding any character other than '/' to a name will not get a sub-part, but instead get a 'brother' of that previous name.

4. These blocks of data are atomic. You can create a new one and nobody sees that fact until you close it and they then open the same name. You can modify one and nobody sees that fact until you close it and they open it. If they have already opened the file they will see the old data until they close it.

5. There are calls to insert data into the blocks as well as overwrite it. In fact it may be ok if insertion and truncation are the only write operations.

This is NOT what Microsoft is going to come up with! Why? Because it is too simple to emulate this on other systems and thus remove any uniqueness of Windows. They do not want this and they will not make it. Because this is not what they are going to make, though, they cannot say they are making the "holy grail" without lying.

Re:Finding your photos

[henryhbk]

Uh, use something like iPhoto, Cumulus, etc... These products already do this, without imposing any changes to the file-system. But the discipline to actually do the keywording is hard (I have about 8000 pictures from my digital camera, and at best, I have a keyphrase for each roll shot)

Re:Data labelling and ACID complience seem quite g

[DavidTC]

You can get infinitely close to ACID complience incredibly easy on any OS.

You just need a program that renames the old version of the file, saves the new version, and then deletes the old.

At worse, you end up with no file by that name, and you have to hunt down what it renamed the old one as.

Any program that can possible cause you to lose your old (and new) data when saving new data is just broken.

To keep linux out

[Stevyn]

The linux people got FAT32 to work on Linux. Then MS comes out with with NTFS, as I remembered from when I tried to get linux to mount NTFS drives, it could only read them. There were things I could add to my linux to get it to write to NTFS, but the developers said to use them with caution. In a couple years, the linux people will figure it out and Linux and Windows will coexist better. But then this WinFS will come out. The linux people will have to go through the whole process. I'm not saying that WinFS will be just another buzz word, I'm sure it will work better than NTFS as NTFS works better than FAT32. But since when has microsoft been about making things work better that the average user wouldn't appriciate? They spent hundreds of millions on IE and WMP because a lot of people use those programs. But John H Computer user isn't going to make his buying decisions on the file system used. I think they're just doing this to lock out their competetitors, i.e. Linux. And they could also through some incredibly simple copywrite or encryption in it that offers no security, but allows them to use the DMCA as a shield against linux the people developing linux support for it.

This has been done already...

[lga]

...a very long time ago.

I used to have an ADDS Mentor computer. It ran the PICK operating system and had a database as the file system. The commands it used were so obscure that it was the only computer where I have ever needed to refer to the manual for basic operations but it was fast enough to run 16 users (terminals) on only a Z8000 at 8MHz with 512K of memory.

This is definately not a Microsoft "innovation."

2 life changing words...

[Dracolytch]

There are two words that describe why it is, in fact, a really BAD idea to integrate SQL with a file system: drop all; ~D

File system recipe

[B5_geek]

2 parts FAT file structure
1 part NTFS for security
1 pinch of FAT32 (to keep the flavour)
1/2 part SQL (so Suzie Soccer Mom can use it too)
4 parts XML to keep DRM in tight control
2000 parts OS bloatwear to understand it

The open source community can do better:

[master_p]

SQL and XML is not needed. All that is needed is for the system to be able to know what each file is and what values it contains:

Each file should be an object. The O/S will maintain each object's class. This class would be able to tell the outside world about its content by full run-time type identification.

For example, let's say I want to find all the bitmaps that are of 16-bit color depth, of size 320x200 to 640x480 and created in September. The system will search the filesystem and check:

-the file's class; if the file is derived from 'Bitmap' or is a 'Bitmap' then it will be processed
-the bitmap's size
-the bitmap's color depth
-the creation date

If a file is elligible for this query, then it will be shown to the user.

It's very simple, and I have talked before about it:
http://slashdot.org/comments.pl?sid=79683&cid =7042 606.

(and I have been completely ignored...)

I think...

[Eric Damron]

"accoring to Microsoft, will open up a whole new world of information availability."

I think that Microsoft has made my personal information available enough to hackers. Thank you very much.

This article is old news

[Overly Critical Guy]

We knew WinFS wasn't a replacement for NTFS over half a year ago. Of course, Slashdot is just now getting around to pointing it out (after previously posting about NTFS being "replaced"), but still.

Perhaps I'm misunderstanding what this will do.

[johneee]

Let me know if this is what we're talking about here...

But it would seem to me that something that would allow me to browse to, say, a Flash MX file by either going:

My documents/clients/client 3/source files/flashThing.fla

or

My Documents/Macromedia Files/Flash MX/flashThing.fla

or

My Documents/creative things/Reusable things/swirls/flashThing.fla

or, or, or...

Without having to have a bunch of shortcuts OR multiple copies of a file... Well, I think this would be an extraordinarily cool thing which would save me oodles of time.

That would be a killer app.

Re:Thoughts on XML - lucid parent post

[MattRog]

Very very good! A thinking practitioner!

When only machines are talking it really doesn't matter WHAT the format is -- preferably it is unambiguous and somewhat speedy to transmit and parse.

As I said in another post - I seriously hope that the data is stored in a DBMS table and the XML-ized version is one (out of many) way to retrieve it.

This is almost what I've been waiting for

[catscan2000]

I've been thinking a lot about data storage over the past couple of years and even spoke with several colleagues on creating a clustered OS with a clustered data store that supports multiple forms of data storage retrieval, including but not limited to hierarchical and relational. Since building an OS from scratch might not be the most practical thing at this time, from my current thinking at least, I'm going to move it to the Application layer to create a data store inside a framework and application that can be a drop-in replacement for Microsoft Access but be multi-platform and run in Java. But if Microsoft's new filesystem starts to take off, then perhaps the work done on this system could be put into the kernel level at that time. Just a thought, though things might get complicated trying to run Java code in the kernel ;-) (unless if the kernel is Java-based). Just some thoughts, now I have to get back to work..

Warning: contents may be dangerously lukewarm.

[hypnagogue]

There should have been a warning with this article to set an alarm clock before reading it.

Now I'm late for my 3 o'clock Friday ditch-out, and my keyboard soaked in drool.

Ye gods...

[Tore S B]

"This release is going to be driven by technology, not by a release date. Which probably means it is going to be late." And in other news... Microsoft announced that Linux had a point and that work on re-releasing longhorn as GPL is underway. The PDP-7 Restoration Project http://tore.nortia.no

History repeats itself...!

[Tore S B]

"I can't think of desktop applications where you would need more than 4 gigabytes of physical memory, which is what you have to have in order to benefit from this technology. Right now, it is costly."

Why did that sound frighteningly like "4GB RAM ought to be enough for everyone"?

DOES THE MAN NEVER LEARN?!?

The PDP-7 Restoration Project
tore.nortia.no

balancing trees

[demo9orgon]

I think some file systems are trying to do too much. Anyone remember the question about balancing binary trees in their data-structures class? (and yes, those of you who haven't had it, use your imaginations)

"What is the best method for balancing a binary tree of increasing size and complexity?"

And the answer,
"There isn't one. The effort is better spent by saving the information in a linear format and rebuilding the structure for the express purpose of solving the immediate problem."

The same thing may apply to a file-system of ever increasing feeping creaturism (a Larry Wall term), size and complexity.

We already keep duplicate copies of FAT and INODE-isms. I can only see adding XML, and NTFS wackyness to the mess as an equivalent to sprinkling powdered sugar on burnt toast. The end result may be a "do-everything" filesystem, but when you get past the tasty topping you can't help but notice things are not so pleasant...over time it will just get worse. Yes, entropy happens in file-systems just like everything else.

Could you imagine your file-system flapping like a router as it's trying to re-index itself? Meanwhile, you just keep clicking on the "Yes" button to save a document and it keeps throwing you blocking dialogs in some kind of race-condition loop.

Oh no, _that_ could never happen. :-D

filesystem != database

[js7a]

Filesystems are just inefficient, shitty databases.

Not really, and you know it.

Firstly, your OS's filesystem should support reading and writing files at nearly the throughput of your disk interface, with efficient buffering and interrupt-driven hardware I/O. Databases do that rarely, and only on the high end, and even there only after you've taken significant configuration pains.

Second, you can have files from 0 length to N gigabytes with the limited overhead of small cluster sizes. BLOBs aren't stored nearly as efficiently in your database.

Third, your database doesn't let you do random seek()s inside your BLOBs.

Fourth, you can't append to a BLOB without reading the whole thing to memory and then storing it back, which means time and bandwidth on a network, unlike NFS/SMB/AFS/etc.

Fifth, you can't keep a BLOB open for writing like a log file.

Sixth, you can't make subdirectories out of a table of key strings and BLOBs without stupid filename hacks, which themselves are likely to break any semblance of wildcards that you may have expected from pattern-matching strings.

I could go on (e.g. device nodes on unix, names pipes, etc.) but you get the point. Although parts of filesystems resemble databases in the abstract, they are different for different purposes.

Ah, Microsoft...

[Illbay]

...Still purveying "vaporware" after all these years!

I thought

[ShadowRage]

ext2/php/mysql/*nix was already a holy grail for developers due to its low price of $0

Thanks for clearing that up CNET!

RE: worth undertaking?

[King_TJ]

I don't know.... When I first read about this, I thought it was a great concept (if done properly). But as I think about it more,I'm not sure it's worth the effort to develop at all.

It's exactly the type of thing a company like Microsoft would see good reasons to develop - but more for marketing reasons than true performance ones.

I can see how making the file system into some type of relational database could offer more options for searching. I can't see how it would make file saves any faster than they'd be without it. I also think it could make recovery of lost/damaged files much more complicated than it already is. (Imagine a scenario where you have to have the skills of a DBA just to get back your accidently deleted file. I'm not sure it'd be as easy as running Norton Utilities and clicking on "repair disk"....)

It also strikes ne as somewhat relevant that 3rd. party tools and document formats have advanced considerably in recent years. Not THAT long ago, people longed for technology that would let them search for documents containing specific text, rather than just for filenames. Now, we've got numerous tools and options for accomplishing that. (That's what "Document Management Systems" are all about, after all.) We can even text search inside of documents that are essentially a graphics snap-shot of pages using Adobe Acrobat.

WinFS needs Windoz to be running. Linux lock-out !

[rickst29]

OpenOffice.org suite in the Windows version ( running on Windows) will theoretically be able to get at the data using 'compatibility' features. But, it will appear to be absolutely impossible for a PC running Linux or another non-Longhorn Operating system to manipulate data on a 'WinFS' filesystem, in the ways that I frequently manipulate data on my FAT32 partition from Linux Applications.

I believe that the main purpose for introducing WinFS is to lock data into Windows computers, requiring Windows to be running in order to have any access at all. The benefit this can provide is to further complicate migrations from Microsoft's OS monopoly environment. I think that it would bring almost no visible benefit to customers. I feel that the vaporware 'benefits' being presented in these presentations are largely the responsibility of user-level programs. For example, if photos are going to be indexed by people in the photos, most of the work to implement this feature must occur in an application to accept the definitions of 'people' (e.g., names, recognizable characteristics) then try to 'recognize' them within other graphic documents.

ISVs will have a miserable time, spending huge $$$ to implement support for WinFS (it is definitely not beneficial for them). Anything which increases the costs for competitive ISVs is good for MS. The increased R&D costs, the longer delays to market, and the increased support costs may also allow Microsoft to successfully enter many markets which are currently dominated by ISVs (e.g. tax preparation, photo editing).

And, if we use WordML as an example, Microsoft appears to have no intention of using XML for the purpose of sharing information... a lot of the stuff you want in an MS-Word data file is hidden in BLOGs, totally undefined except for the fact that the elements have names and contain hex-64 data.

Microsoft has usually made excellent financial decisions. Microsoft is spending a great deal of money on this effort, and the additional delay in releasing their new OS will probably cost them a great deal of revenue. I think that Microsoft is doing all of this work to preserve and extend their OS monopoly position.

Re: worth undertaking?

[EddWo]

The recovery of lost damaged files issue is probably why they are going to use NTFS underneath it. Businesses have 10 years of experiance working with NTFS and there are many third party tools.

The database aspect is just what you will see through the shell and the file dialogs. Underneath programs will probably still open and save files the way they always have. New applications that can make use of the database and respond to triggers etc will come online eventually but for backwars compatibility reasons if nothing else the old file system structure must be there underneath.

I expect it won't bother making a database of the entire file system anyway. Why would you want all your system files in there? Just index everything with a user profile and other selected directories like movies, music, pictues etc. Those are the things people spend most time searching for.

Unix better for annoying ex's

[TheConfusedOne]

You can use the 'dig' command. :-D