I guess the quick answer for that is practicality. Just like *NIX people pride themselves that they are allowed to do "rm -rf *", windows people want to be allowed to rename something.exe and run it...
Yes, but those are two seperate problems. It's not wise to say "Ditch the WinFS project because people don't have good meta data anyways".
There are Perl scripts out there that do a pretty good job of cleaning up crummy id3 tags btw... They try and deduce stuff from the file name, and it works relatively well.
Last but not least: I personally tried the WinFS thing and am still in the process of kinda trying it. What I did is import my file listing for the drive onto my SQL db. I also created an extended stored proc that moves a file. I linked this stored proc to the update trigger for my file table. And I also made MD5 checksums for each file (that was a pain). You'd be surprised how much you can get done with that setup.
SELECT F1.* FROM Files F1 JOIN Files F2 on F1.FileName != F2.FileName and F1.Digest = F2.Digest
That right away gives you all the dupes. Instantaneous. Then moving files around is a breeze.
Anyways, the problem with this setup, and also the setup with Symlinks is that when I add files to my folder, they don't get added to the db. So things fall out of synch when I'm lazy. Same would happen for symlinks.
You've got a point about hansreiser. In any case though, his post was aside the point IMHO.
To answer your question though, it's really a case where you actually *can* and Microsoft is the first to *do* 'hide' stuff in alternate streams.
Z:\...\Desktop>streams *
NTFS Streams Enumerator v1.1 Copyright (C) 1999 Mark Russinovich Systems Internals - http://www.sysinternals.com
Failed to open MEMORY.DMP: Access is denied. Desktop\12 Madness in great ones.mp3: :Zone.Identifier:$DATA 26 Desktop\2Funny-Wedding-Commercial-Video-FunFr y_com.mpeg: :Zone.Identifier:$DATA 26 Desktop\3dmax_to_maya.pdf: :Zone.Identifier:$DATA 26 Desktop\ephpod275.exe: :Zone.Identifier:$DATA 26 Desktop\iTunesSetup.exe: :Zone.Identifier:$DATA 26
I'm assuming those identifiers have something to do with the new feature in SP2 that pops up a dialog box if you try to run any file you've downloaded via IE. Regardless of whether you have renamed it or moved it around.
Also, Winrar gives you the option to "Encode alternate NTFS streams"... so it is out there for sure.
It's funny because the only reason this feature isn't being exploited more is probably because of standard hacker mentality that Windows isn't a system worth exploring. I can tell you, alternate streams would be a mean vector to spread a virus on.
Yeah, but check this out: I have about 90 gigs of mp3s. Perfectly organized, (artist\album\artist - track number - title.mp3). But there is no way I can find mp3s by genre, year, or even album.
That's why I need to be able to 'morph' my directory structure simply with a statement: say "GROUP BY Year", or "GROUP BY Genre, Artist" or whatever.
It seems anyone here on/. right away thinks of complex join operations when the word relational is mentionned, I'm only thinking of relatively simple SQL statements that return instantly. That might be a step down from what relational dbs are meant to do, but it still is a heck of a lot more than what file systems are capable of today.
I'm not so convinced about this: NTFS has things called streams. Security descriptors are in an alternate stream for example.
You can have any amount of streams on a file.
The only downside is that it's not 'officially' supported, for example, the filesize reported isn't consistent with all the streams (only the $DATA stream is used to compute filesize). That being said, it's not a hidden API either, it's just a question of opening a file name with the stream specified.
Long story short though, you can put any sort of meta data you want in alternate streams. In fact, Microsoft does this quite often in their own files.
First off, Access has been on the dead-man-walking list for a LONG time now. They've been meaning to scratch that piece of shit for a while. (Did you know that AD was based on Access - I don't know if it still is).
Anyways, I think that beyond not having answers, it's just a bigger task than they envisionned. Requiring probably advances in areas that they never thought of: one thing I can think of right away is that MS SQL code is mostly C++ code. And C++ code doesn't work in the kernel (all the C++ exceptions etc don't mingle well with the kernel ways of doing things).
So they might have been forced to rewrite stuff... etc. etc.
And that's why it's taking so long. Accessing filesystems as SQL data has always been a dream of anyone who has had many files. They just never knew about it.
WinFS is the 'real' solution IMO to all things like iTunes playlist managers, and expensive Content Management Systems yadi yada.
Sure, no consumer is expected to actually use SQL statements, but that doesn't mean that user mode programs should *implement* SQL features. User mode programs should only be the 'translation' layer between the user's point and click GUI, and the OS' internal implementation of the db. Surely, anyone can see that collecting meta data from the file system, and duplicating it in usermode so that you can have search capabilities on it is wasteful.
This article wasn't news to me, I've actually been waiting for this damn WinFS since just about 1996... And by god, is it ever turning into Duke Nukem Forever, but you know what, it's such a cool feature that I still can't wait for it to come out... (figuratively speaking)
I understand your point, but I personally believe that would be a step backward... Especially since in a fully interruptible, fully pre-emtible environment, you don't really have a 'context'.
What I mean by that, is that when the ISR for the video driver fires, the processor is just simply put at that position in code, there is no context switching that occurs... and so you might be running in the thread context of user Joe's command line... that means that there would be no way of determining 'where in the kernel' you are, or whose ISR is firing... or even if it *is* an ISR at all, and not a Dispatch routine (DPC).
Now, I personally believe kernel code should not throw an uncought exception: it is an indication that the code behaved in an unexpected way, and thus an indication that the kernel might not be stable anymore. Really though, it doesn't matter what I believe - it seems the honchos at microsoft believe this too. I personally am happy with it: I'd prefer a BSOD and have to reboot than rebooting a month later to find out my MBR and all files I've written for the past month have been irrecoverably corrupted by an unstable kernel.
The Windows kernel structure isn't like that. A bug check, or BSOD isn't because something has erased the entire kernel, it's because something *might* possibly have begun to go amiss... It is that policy that makes that an uncought exception by a any code raises a BSOD. As well as running at an IRQL that is not appropriate. In fact, those two reasons are the most common BSOD reasons.
And yes I'm aware of the way some code behaves in linux, and I personally am happy about the fact that you cannot be allowed to seg fault in kernel code. That's a disaster waiting to happen man.
On a side note: is it still that way in linux now that the kernel is pre-emtible/interruptible (like NT has always been) ?
It would cost a gaming company almost *twice* the money to implement a game for two operating systems. They may be able to avoid coding everything twice, but they gotta make sure it works twice (QA), and they gotta make sure it's compatible and the same looking on both.
The graphics library doesn't have as much to do with it as you might think.
To add to that: because they started with DirectX 1, and they are at version 9. Had they tried to do something like that with OpenGL, they would have been shat on by any openstandards advocate for "embracing and extending".
Why? Because DirectX is a perfectly published set of interfaces. You are saying, "why isn't ford building Mazda's cars for them, since they invented the fucking car, they should at least have the courtesy to build and deliver to others"...
Is that your point? or did you mean something much more realistic?
. It drives all hardware development. There is no hardware built for SDL or OpenGL or what have you. Every feature--bad or good--is built to conform to Microsoft's latest DirectX rather than the other way around. And you thought Microsoft's monopoly was just in software! HA! And wait til you see what Longhorn itself brings to hardware!
Buddy, that's just a flat out lie. DirectX 9 was a 'save' made by microsoft to actually let programmers be able to use all this radically new techonology that nVidia and ATI both on their own terms created.
Do you not remember the days on slashdot when articles were being posted on how nVidia was claiming they could now render real time cinematics? All those articles were based on a new techonology that nVidia created on its own.
The perfect proof that DirectX is *not* driving hardware development is the radically different structures in ATI and nVidia cards.
I agree with your other points mostly... Except about change. DirectX changing *is* a good thing specifically because it is perfectly backwards compatible. Every new release, there are features programmers have been salivating over in their dreams that becomes accessible to them.
My source would be what the Microsoft PR guy is stated to say in the article:
One of the first orders of business is to "fix busted stuff," as Blythe put it. These items include no more blue-screens (hard crashes) caused by the graphics driver, and moving more processing into what's known as user mode.
[...]
To that end, Microsoft is investing considerable development resources into ensuring that crashes will be very rare, and that when they do occur (and they will), the graphics subsystem can do a snap-reset so that the user will hopefully not be aware that the subsystem even had a problem.
It's in the article man. One of the most important points made actually. I don't know how much of a divergance there can be from such a statement. If they say "no more graphics driver related BSODs", the *only* possibility is that the driver now sits on ring 3...
after 7 years, microsoft still doesn't provide opengl as an official api, as requested by developers
Wtf?? After 12 years, Linux still doesn't implement the win32 API. So fucking what? Oh, and on that same note, does any other platform implement DirectX (aside from wine)? No. So why should Moft implement other people's APIs?
I know about how it used to be, but if you read the actual article, it says they are moving it to "user-land"... hence my saying "I really want to see how this is going to turn out, what with Graphics being an uber high bandwidth thing and all..."
How much more explicit can it be man? Give me a break and RTFA.
On a side note, I am sure in the 15 years since NT 3.5 came out, there has been some developments in the kernel, and programming know how to allow for better performance than what we had back then.
Man, today, it doesn't matter what API you use... The days of backward archane APIs are gone. Every API is just as good, and you know why? Because every API has the goal of allowing access to the underlying structure of the GPU.
Besides, developers today aren't 1 man teams pent up in their basements working against Big Brother, they are billion dollar industries (EA, id, whatever...) who have top of the line programmers who could make *any* API work regardless (because they have the budget to do so), who only really care about the performance and capabilities afforded by the API. Microsoft - like any other big company tending a big market - tries to please them, not piss them off!
IMHO, the time of the underdog syndrome is past... Let people use whatever friggin API they want. It's not like the gaming industry is in the middle of a standards battle.
On a different note, the really amazing thing about Avalon, and you gotta commend Moft for this, is that they're actually moving the graphics driver to User-mode. Just imagine what a gi-nourmous task that is... Let's you appreciate how they can have so much programming going on in there.
You could say the same thing about the 'millions' of people reading the printed version of Encarta's mistake. Imagine also the millions upon millions of people who will continue reading that because nobody buys a new encarta every year.
More than the timespan of a mistake, IMO the bigger problem is the sheer number of mistakes there can be at any one given moment. Making the whole thing lose credibility.
What's worse, is that's a psuedo-OTP via obscurity
Just as much as SSH is a encryption protocol by obscurity: you need to hide the key. If the synchronization timestamp of the communication is hidden, it is the best OTP you can ever get. It comes down the basics of cryptography: you need at least one secret to protect. And that's the weekest link in the whole chain.
Sheer amount of data
is the only reason why SSH and any type of encryption works today: because it would take billions of years to crunch through the possible data set (of a single 128 bit key).
So if a satelite is broadcasting at say 1 MB/s, purely random data (not pseudo-random), and you want to decrypt a 1 meg file by brute force:
First off, you can't just try and decrypt the first 10 bytes to see if you've got the correct sequence: it's an OTP, so only the first 10 bytes might be right, the rest might be wrong.
As a corrolary to that, you have no real way of verfying that what you have decrypted is that actual OTP, since you could coincidentally arrive on an OTP that reveals just parts of the plain text, but not all.
Second, in just one hour 3.6 Gigs of data is generated... that means you need to decrypt the 1 Meg file 3.6 billion times. Assuming you have recorded the stream.
It's as good an OTP as you'll get man.
A bit OT, but no less interesting point...
on
The End of Encryption?
·
· Score: 2, Interesting
I don't remember whether it was an Arthur C. Clarke invention or not, but there was a way to have a perfect one time pad...
It is by having a satelite system, like the GPS system, constantly broadcast synchronized truly random data. That way, any two recipients can communicate as much data as they want by simply synchronizing their communication to a certain time, and using the one time pad that's being broadcast by the satelites at that point in time.
Now, this turns the one time pad bandwidth issue into a secret key (the timestamp) issue. How do you communicate the timestamp between the two sources?
There's another benefit though, and that's that the sheer amount of data being generated by these satelites will make it prohibitive to analyze a transmission after the fact... only live interceptions could be used.
Slashdot Mirror
It must be vacuously true then.
At least I do. It's useful at times.
There are Perl scripts out there that do a pretty good job of cleaning up crummy id3 tags btw... They try and deduce stuff from the file name, and it works relatively well.
Last but not least: I personally tried the WinFS thing and am still in the process of kinda trying it. What I did is import my file listing for the drive onto my SQL db. I also created an extended stored proc that moves a file. I linked this stored proc to the update trigger for my file table. And I also made MD5 checksums for each file (that was a pain). You'd be surprised how much you can get done with that setup.
That right away gives you all the dupes. Instantaneous. Then moving files around is a breeze.Anyways, the problem with this setup, and also the setup with Symlinks is that when I add files to my folder, they don't get added to the db. So things fall out of synch when I'm lazy. Same would happen for symlinks.
To answer your question though, it's really a case where you actually *can* and Microsoft is the first to *do* 'hide' stuff in alternate streams.
I'm assuming those identifiers have something to do with the new feature in SP2 that pops up a dialog box if you try to run any file you've downloaded via IE. Regardless of whether you have renamed it or moved it around.Also, Winrar gives you the option to "Encode alternate NTFS streams"... so it is out there for sure.
It's funny because the only reason this feature isn't being exploited more is probably because of standard hacker mentality that Windows isn't a system worth exploring. I can tell you, alternate streams would be a mean vector to spread a virus on.
That's why I need to be able to 'morph' my directory structure simply with a statement: say "GROUP BY Year", or "GROUP BY Genre, Artist" or whatever.
It seems anyone here on /. right away thinks of complex join operations when the word relational is mentionned, I'm only thinking of relatively simple SQL statements that return instantly. That might be a step down from what relational dbs are meant to do, but it still is a heck of a lot more than what file systems are capable of today.
You can have any amount of streams on a file.
The only downside is that it's not 'officially' supported, for example, the filesize reported isn't consistent with all the streams (only the $DATA stream is used to compute filesize). That being said, it's not a hidden API either, it's just a question of opening a file name with the stream specified.
Long story short though, you can put any sort of meta data you want in alternate streams. In fact, Microsoft does this quite often in their own files.
For more info, check here.
First off, Access has been on the dead-man-walking list for a LONG time now. They've been meaning to scratch that piece of shit for a while. (Did you know that AD was based on Access - I don't know if it still is).
Anyways, I think that beyond not having answers, it's just a bigger task than they envisionned. Requiring probably advances in areas that they never thought of: one thing I can think of right away is that MS SQL code is mostly C++ code. And C++ code doesn't work in the kernel (all the C++ exceptions etc don't mingle well with the kernel ways of doing things).
So they might have been forced to rewrite stuff... etc. etc.
And that's why it's taking so long. Accessing filesystems as SQL data has always been a dream of anyone who has had many files. They just never knew about it.
WinFS is the 'real' solution IMO to all things like iTunes playlist managers, and expensive Content Management Systems yadi yada.
Sure, no consumer is expected to actually use SQL statements, but that doesn't mean that user mode programs should *implement* SQL features. User mode programs should only be the 'translation' layer between the user's point and click GUI, and the OS' internal implementation of the db. Surely, anyone can see that collecting meta data from the file system, and duplicating it in usermode so that you can have search capabilities on it is wasteful.
This article wasn't news to me, I've actually been waiting for this damn WinFS since just about 1996... And by god, is it ever turning into Duke Nukem Forever, but you know what, it's such a cool feature that I still can't wait for it to come out... (figuratively speaking)
What I mean by that, is that when the ISR for the video driver fires, the processor is just simply put at that position in code, there is no context switching that occurs... and so you might be running in the thread context of user Joe's command line... that means that there would be no way of determining 'where in the kernel' you are, or whose ISR is firing... or even if it *is* an ISR at all, and not a Dispatch routine (DPC).
Now, I personally believe kernel code should not throw an uncought exception: it is an indication that the code behaved in an unexpected way, and thus an indication that the kernel might not be stable anymore. Really though, it doesn't matter what I believe - it seems the honchos at microsoft believe this too. I personally am happy with it: I'd prefer a BSOD and have to reboot than rebooting a month later to find out my MBR and all files I've written for the past month have been irrecoverably corrupted by an unstable kernel.
And yes I'm aware of the way some code behaves in linux, and I personally am happy about the fact that you cannot be allowed to seg fault in kernel code. That's a disaster waiting to happen man.
On a side note: is it still that way in linux now that the kernel is pre-emtible/interruptible (like NT has always been) ?
Yup, you're 100% right. =)
It would cost a gaming company almost *twice* the money to implement a game for two operating systems. They may be able to avoid coding everything twice, but they gotta make sure it works twice (QA), and they gotta make sure it's compatible and the same looking on both.
The graphics library doesn't have as much to do with it as you might think.
(sorry if I appeared standoff-ish, I was responding to the hords of knee-jerk anti-microsoft replies).
Why? Because DirectX is a perfectly published set of interfaces. You are saying, "why isn't ford building Mazda's cars for them, since they invented the fucking car, they should at least have the courtesy to build and deliver to others"...
Is that your point? or did you mean something much more realistic?
Meanwhile, jackasses that make totally useless points, about how OpenGL should have saved us from Ebola and the Plague get moderated 5+ Insightful.
I agree so 100% with your point that I just have to quote a chapter title from a book which I use as a development bible... it goes:
Portability is for canoes
Buddy, that's just a flat out lie. DirectX 9 was a 'save' made by microsoft to actually let programmers be able to use all this radically new techonology that nVidia and ATI both on their own terms created.
Do you not remember the days on slashdot when articles were being posted on how nVidia was claiming they could now render real time cinematics? All those articles were based on a new techonology that nVidia created on its own.
The perfect proof that DirectX is *not* driving hardware development is the radically different structures in ATI and nVidia cards.
I agree with your other points mostly... Except about change. DirectX changing *is* a good thing specifically because it is perfectly backwards compatible. Every new release, there are features programmers have been salivating over in their dreams that becomes accessible to them.
One of the first orders of business is to "fix busted stuff," as Blythe put it. These items include no more blue-screens (hard crashes) caused by the graphics driver, and moving more processing into what's known as user mode. [...]
To that end, Microsoft is investing considerable development resources into ensuring that crashes will be very rare, and that when they do occur (and they will), the graphics subsystem can do a snap-reset so that the user will hopefully not be aware that the subsystem even had a problem.
It's in the article man. One of the most important points made actually. I don't know how much of a divergance there can be from such a statement. If they say "no more graphics driver related BSODs", the *only* possibility is that the driver now sits on ring 3...
Wtf?? After 12 years, Linux still doesn't implement the win32 API. So fucking what? Oh, and on that same note, does any other platform implement DirectX (aside from wine)? No. So why should Moft implement other people's APIs?
How much more explicit can it be man? Give me a break and RTFA.
On a side note, I am sure in the 15 years since NT 3.5 came out, there has been some developments in the kernel, and programming know how to allow for better performance than what we had back then.
Besides, developers today aren't 1 man teams pent up in their basements working against Big Brother, they are billion dollar industries (EA, id, whatever...) who have top of the line programmers who could make *any* API work regardless (because they have the budget to do so), who only really care about the performance and capabilities afforded by the API. Microsoft - like any other big company tending a big market - tries to please them, not piss them off!
IMHO, the time of the underdog syndrome is past... Let people use whatever friggin API they want. It's not like the gaming industry is in the middle of a standards battle.
On a different note, the really amazing thing about Avalon, and you gotta commend Moft for this, is that they're actually moving the graphics driver to User-mode. Just imagine what a gi-nourmous task that is... Let's you appreciate how they can have so much programming going on in there.
They seem to be moving the graphics drivers into user-mode.
I really want to see how this is going to turn out, what with Graphics being an uber high bandwidth thing and all...
More than the timespan of a mistake, IMO the bigger problem is the sheer number of mistakes there can be at any one given moment. Making the whole thing lose credibility.
Just as much as SSH is a encryption protocol by obscurity: you need to hide the key. If the synchronization timestamp of the communication is hidden, it is the best OTP you can ever get. It comes down the basics of cryptography: you need at least one secret to protect. And that's the weekest link in the whole chain.
Sheer amount of data
is the only reason why SSH and any type of encryption works today: because it would take billions of years to crunch through the possible data set (of a single 128 bit key).
So if a satelite is broadcasting at say 1 MB/s, purely random data (not pseudo-random), and you want to decrypt a 1 meg file by brute force:
First off, you can't just try and decrypt the first 10 bytes to see if you've got the correct sequence: it's an OTP, so only the first 10 bytes might be right, the rest might be wrong.
As a corrolary to that, you have no real way of verfying that what you have decrypted is that actual OTP, since you could coincidentally arrive on an OTP that reveals just parts of the plain text, but not all.
Second, in just one hour 3.6 Gigs of data is generated... that means you need to decrypt the 1 Meg file 3.6 billion times. Assuming you have recorded the stream.
It's as good an OTP as you'll get man.
It is by having a satelite system, like the GPS system, constantly broadcast synchronized truly random data. That way, any two recipients can communicate as much data as they want by simply synchronizing their communication to a certain time, and using the one time pad that's being broadcast by the satelites at that point in time.
Now, this turns the one time pad bandwidth issue into a secret key (the timestamp) issue. How do you communicate the timestamp between the two sources?
There's another benefit though, and that's that the sheer amount of data being generated by these satelites will make it prohibitive to analyze a transmission after the fact... only live interceptions could be used.