That's mostly right, but I think the kernel's role is a bit smaller.
Tokens, which identify the access that a process gets, and access checks against kernel objects with security descriptors are indeed handled by the kernel. Integrity levels attached to tokens and mandatory integrity labels are new properties that the kernel allows for tokens and security descriptors and consults during an access check in Vista. That's really the extent of kernel involvement for integrity labels.
The decision of what to put into a token or security descriptor has always been up to user mode components. Winlogon uses a restricted, medium integrity version of an administrator's token for starting the shell. IE launches a low integrity child process of itself. Certain functions, like Win32's CreateProcess check the image manifest for requestedExecutionLevel and enlist the seclogon service if needed to elevate the new process. The kernel syscall to create a process doesn't do any of this.
Windows messages are implemented by the Win32 subsystem, not the kernel (even though part of Win32 does run in kernel mode in win32k.sys). The kernel does supply several IPC methods, but not this one. Win32 does the target window integrity check (UIPI).
I would say that UAC is a user mode construct, mainly implemented in winlogon, kernel32.dll, LSA and the seclogon service. The kernel does enforce restricted tokens and integrity labels for kernel objects though. I can't say that the NT security model has been followed as well as it could be (part of this is due to its complexity), but Vista (and UAC) do use a lot more of it, and seem to have avoided serious security or compatibility problems. One issue is that NT was designed for use on a LAN with trusted programs and the security system was for protecting users and the system from unauthorized users, not from malicious programs. Thus, the owner of a computer is the Administrator, since the system doesn't have anything that user shouldn't have access to. Unfortunately, this doesn't work so well when that admin can't trust all the programs on the system to behave.
I was thinking of one transaction, where the file is truncated and new contents are written. Either both entirely happen, or neither do.
OTOH, I think you're on to something with a simplified transaction system that allows atomic pairs of operations that can be used to implement things like this, e.g. replace one file with another, ensuring it's written completely on commit.
Right click on your profile, go to properties, advanced, check the encrypt box. Alternatively, cipher/e/s on your profile dir. 5 seconds of googling surely would have revealed this.
It sounds like the correct solution is for the file system to implement transactional semantics. That is what the applications need and were incidentally getting, despite it not being in the spec.
Why isn't this being considered as the solution? There are other major OSes have implemented basic atomic transactions in their filesystems successfully, why not Linux?
Then you have some other corruption problem that has nothing to do with the layout of user profiles.
Besides, I didn't have to google to determine the layout, I could see it after a minute of browsing. Google provided me with more formal documentation to reference in my post.
I never said there weren't other problems. I consider the default name for those paths to be hideous. Plus with the ridiculous and horrible limitation that is MAX_PATH in so many Win32 components, that is path real estate that we can't afford to spend. Parts of Win32 go to great lengths to work around MAX_PATH, including searching for shorter surrogate paths to use instead..NET, foolishly built on Win32, has the same problems.
At least they changed the defaults to "Users", "Documents" and "AppData", etc in Vista.
I was responding to the specific complaint that profile paths were arcane and not understandable.
Makes sense that the Outlook data would be in C:\Documents and Settings\\Program Data\Microsoft\Outlook but it's not.
Instead, the roaming stuff goes into: C:\Documents and Settings\USERNAME\Application Data\Microsoft\Outlook And the non-roaming stuff goes into C:\Documents and Settings\USERNAME\Local Settings\Application Data\Microsoft\Outlook Doesn't seem so awful.
The only way to ehfin find it is to back the stuff up! What if the computer crashed and I can't RUN outlook???? I'm hosed (this actually happened)
What about the Windows Credentials Management subsystem? It's been there since XP. IE and Explorer use it for the remember password option. The list of usernames/passwords in your profile can be modified via the users control panel.
However, Microsoft does seem to prefer a single (or very few) signon system with an AD domain or Passport.
If mass storage did not have the HID abstraction and wear levelling circuitry (primitive though it may be), Windows would have absolutely soiled every flash device out there with its uniquely bad IO layer.
Can you be more specific as to Windows' "uniquely bad IO layer"?
Block allocation is the responsibility of the filesystem. Windows doesn't have a flash optimized filesystem because it would 1. break backwards and cross compatibility because MS would have to implement a new filesystem, one that they wouldn't port to previous OSes and wouldn't be compatible with other OSes, either because of NIH syndrome or because other OSes don't have a raw flash optimized FS (i.e. OSX) and 2. as the parent said most consumer hardware does not expose the raw blocks to the interface, so the FS would be of limited value.
However, there is a lot more to Windows' IO layer than filesystems, and there's nothing in the rest of the IO system preventing a raw flash optimized stack. I think Microsoft considers this to be a hardware problem, best solved in hardware, where it has been solved in hardware.
NT was written from scratch. At one point it was going to replace the old OS/2 codebase, but the breakup happened and it ended up replacing the old Windows codebase (albeit after the old line had 3 more major releases).
I was under the impression that modern Linux kernels had negligible performance impact from using a swap file as opposed to a dedicated swap partition.
Personally, I much prefer using a swap file because it gives me more flexibility in locating, resizing and moving swap.
If felons were barred from serving in the Senate / Congress, then all you'd have to do is get people you don't like convicted of little petty things which may technically be felonies (due to their positions) and you could control who was in charge.
Windows NT was designed to be multi-user from its first release. Groups and privileges have always been provided for role based security.
I, for one, am glad they didn't copy a 1989 standard unixy security model with limited UGW/RWX, root or nothing privilege, primary groups, multiple object namespaces, NFS problems, etc.
Users can rearrange their own start menu and desktop without a UAC prompt or affecting the system. Users can't modify the All Users common profile that affects the start menus of all users on the system.
Explorer displays items from both locations side by side, so this isn't obvious. NT4 used to have a different icon for common items.
Many installers just dump their items into the All Users profile, which makes modifying it a pain.
Doesn't the Vista installer create a UAC administrator account that needs elevation to exercise admin privileges? Sort of like the OSX default admin account?
In none of those situations will you get ANY benefit at all from disk caching... yet if I watch a 4 GB movie over a period of 2 hours, a lot of memory managers will decide that attempting to cache all of that data might be a good thing.
That's what cache hints are for. The media player should be opening the file with a hint that it will be read sequentially. On NT, this means that the cache blocks go to the end of the list, the first to be recycled.
Another problem is that the kernel doesn't know what data is important and what isn't. In Vista (and WS2008), there are 8 cache priorities, which helps fix this. Background processes run at the lowest priority and can't encroach on the high priority memory used by interactive applications.
Haskell programs can use whitespace to define blocks. However, anything that can be done with the layout rule (whitespace) can also be done with braces and semicolons, so you can have it either way.
Actually, the database backend for Exchange is the Extensible Storage Engine, previously Jet Blue. This is the same engine that Active Directory uses. ESE is an entirely different engine than Access uses, sometimes called Jet Red. ESE is just a table engine: it does not provide a high level interface like SQL. Jet Red has best-effort error recovery. ESE uses a journal.
winerror.h is also shipped with the WDK. It contains all the error codes for the Win32 environment subsystem, which isn't implemented by the kernel. Win32 exists mostly in user mode: although a sizable chunk of the Win32 server process was moved from winsrv.dll into win32k.sys in NT4, which does run in kernel mode, Win32 really isn't part of the kernel. They did that to cut down on IPC overhead, not to integrate it into lower layers. Win32 services can only be used by user mode processes.
ntstatus.h is the header that contains the kernel's error codes. That file is 16272 lines in the WDK 6000 (with 8 comment lines to every code), and contains few codes not relevant to the kernel.
It's not surprising that winerror.h contains messages not related to the kernel because it's not what I'd consider part of the kernel. AFAICT, this file is included only to support win32k.sys, and possibly display drivers. I'd be interested to know how many of the kernel source files actually include winerror.h.
I do agree that this problem has been solved for a long time, but I think it's done better with modern functional languages. I'm looking at Limbo's features, and it looks like GHC does all of those things, and more, better. Take the multithreaded "Program 2" example from that Limbo example page, implemented in Haskell:
module Timer where import System.Environment import Control.Concurrent import Control.Concurrent.Chan
main = do { putStrLn "Command Line Parameters"; sync <- newChan; args <- getArgs; forkIO $ timer sync $ length args; sequence_ $ zipWith (zipf sync) [1..] args; } where zipf sync n s = (readChan sync) >> (putStrLn $ (show n) ++ ": " ++ s)
timer:: Chan Int -> Int -> IO () timer sync n = mapM_ (\n -> (threadDelay $ 1000000) >> (writeChan sync n)) [1..n]
Haskell's not even a sequential language by nature. Standard Haskell code is a series of side-effect free, deterministic expressions that are executed on demand, which makes execution timing issues (even across CPUs) much less relevant (and invisible to data processing in the program).
Just to clarify, NtCreateProcess, the kernel syscall to create a process, is much lighter than Win32's CreateProcess function. NtCreateProcess doesn't even start any threads in the new process or load dependent libraries. CreateProcess includes out of process calls to register with CSRSS, an application compatibility database lookup since XP (so slow that there's a special option on WS2003 to turn it off), tons of registry reads, among other things.
NtCreateProcess even supports COW fork, by specifying a NULL image to load-- it's the Win32 libraries that aren't designed to support forking. POSIX^W SFU^W SUA processes are much lighter than Win32's and can use the kernel's fork support.
An IRP is an IO Request Packet. They absolutely were used in every version of NT, including 3.1.
A DPC is a Deferred Procedure Call-- similar to the bottom half of an interrupt handler on Linux. Those were also definitely in all versions of NT. WDM just provides a wrapper over such functions.
At no point did NT use 16-bit code in kernel mode.
OS2/NT was being written from scratch, but then Microsoft and IBM parted ways and it became Windows NT.
NT 3.51 didn't introduce a new driver model. It did introduce FastIO, which is a function call oriented alternative to issuing IRPs for certain synchronous IO operations (particularly those that can be done from cache).
Modern CPUs implement virtual memory, which means that any given page of memory (the smallest unit of allocation, per the CPU hardware) may not be directly accessible. The OS can decide to move a page to disk to make room for other things, or implement a memory-mapped file. When that page is accessed, it causes a page fault, which the OS silently fixes by moving the page back into memory. The memory currently accessible without causing a page fault and OS intervention is the working set. The amount of memory allocated (i.e. the amount of storage reserved by the OS), and the working set can be different.
Linux implements a global working set. This means that when the OS decides that it wants to remove a page from its working set (i.e. swap it to disk), it pulls that page out of all the processes that are using it (since the page might be shared), writes it to disk and marks the page as free.
Windows NT implements a per-process working set. A page is moved out of the working set of a particular process, and when it's been removed from all processes (in case it was shared), it goes into the standby cache, a sort of limbo where the page exists both in memory and disk and can't be written to (this makes it possible for the page to be moved completely to disk or back into use without further disk access). Each process has a soft minimum and maximum working set that the memory manager tries to keep a process within. Memory heavy processes have their max working set automatically expanded.
"That is, the Windows Task Manager is reporting the memory cache allotment and not the memory allocated and used by the program. This is not the same as a memory leak. The program has little to no control over the memory cache allotment the OS has given it."
Task Manager reports a process's current working set under the heading of mem usage. More pages allocated to the process may in fact be in memory in the standby list, but they won't show up in this count. Memory cached by the OS (e.g. standby cache, file cache, write cache) is not counted in the working set of any process, even if only one process is really using it. They show up as "System cache" in the performance tab, and some caches are double counted as "Available" because they can be discarded without disk access. When a process calls NtFreeVirtualMemory (the syscall for freeing private memory pages), the OS does not keep it in the process's working set. The working set is always equal or smaller than the sum of shared and private memory allocated to the process. If a process were to free all its private memory, and somehow unload all its modules and free its stack, the working set would go down to zero. A program has full control over what memory is allocated to it. It can't fully control how much of that memory is actually resident in RAM, though, and that's what is reported by Task Manager.
When you minimize a window, the Win32 subsystem sets the process's maximum working set to the system minimum, effectively moving most of its private pages into standby. Those pages will only come back into the working set as they're accessed-- it may take a long time for the working set to get has high as it was previously, and possibly never for memory leaks or unused caches. Firefox definitely implements some hefty caches.
In short: yes Windows implements a memory cache, but not for pages that have been freed.
1. IO priorities. Yes, these are quite nice, although it'd be nice if more than 2 priorities were implemented (currently only background and normal are supported). Memory priorities are new in Vista as well, and are also quite useful.
3. Much of the UMDF has been backported to XP as a redistributable.
6. IO cancellation. The NT kernel has always supported cancellation of pending async IRPs. The kernel always tries to cancel pending IRPs in a dying process so it can be terminated. However, there have been places where drivers would hold on to an IRP for a long time without letting it be canceled, and some of the places where that was happening have been addressed in Vista. A notable case was during a MUP open, i.e. when the system was trying to locate the host and service behind a UNC path. Vista does now allow cancellation of a synchronous IO operation, which previously could only be done by killing the thread.
5. Shell multithreading. I don't agree that Explorer and the shell have become substantially more multithreaded. Synchronous IO is still done heavily in the GUI thread, which causes the window to stop responding. The aforementioned ability to cancel sync. IO in another thread was added to make it possible to abort such blocking, which is a poor workaround.
7. Async IO in SMB (the LanmanRedirector filesystem internally). It has always supported async IO. The new CTRL+C ability to abort is used to cancel synchronous IO in the UI thread. One issue is that some APIs can only be done synchronously, such as file opens. You can only associate a file with an IOCP after the file is opened. From the application perspective, this could be fixed by having worker threads do the blocking open operations in the background.
8. Transactions. Yes, very nice. The transaction engine is even extensible: ISVs can add their own transaction resources (MS SQL server supports this), and the DTC service can coordinate them across multiple machines.
9. VSS put to better use. Well, Microsoft has a bad habit of putting good features into the system and then dropping the ball when it comes to making use of them. It's like the teams don't communicate with each other. Something that XP did use VSS for was in ntbackup, to archive a locked file by copying the shadow version. Vista backup does not supports this, and ntbackup is incompatible with Vista.
In the end, I'm with you: until they fix the things that have been broken in Vista, I plan to stay with XP and 2003 for some time.
To be fair, Windows NT had granular, integrated and standardized security, networking, virtual memory (even in kernel mode), a unified page cache, SMP, and other advanced features right from its first release in 1993. Dave Cutler and the rest of the VMS team MS acquired definitely knew what they were doing.
The exploits that Windows has had have very rarely been kernel or core design issues. Windows has a secure design, but it's rarely configured to be secure and has suffered numerous implementation faults. In particular: usermode components, notably the shell, LSA and RPC. Microsoft is also guilty of putting compatible defaults ahead of secure ones, e.g. making Admin accounts default in XP. OEMs are also partially to blame here because they decide how the computer comes loaded from the factory (i.e. with one auto-logon admin account), network admins for allowing it to stay that way (in a corp environment), and ISVs for making tons of software that requires admin privileges when it shouldn't.
With Vista, Microsoft is trying to keep around as much old code as possible in certain components, maintain compatibility with old software, change the default privilege level for programs to non-admin, and implement some kind of TCPAesque DRM. In short, they're trying to have their cake and eat it too, via technical means. It's not pretty. Time will tell how effective it is.
Slashdot Mirror
That's mostly right, but I think the kernel's role is a bit smaller.
Tokens, which identify the access that a process gets, and access checks against kernel objects with security descriptors are indeed handled by the kernel. Integrity levels attached to tokens and mandatory integrity labels are new properties that the kernel allows for tokens and security descriptors and consults during an access check in Vista. That's really the extent of kernel involvement for integrity labels.
The decision of what to put into a token or security descriptor has always been up to user mode components. Winlogon uses a restricted, medium integrity version of an administrator's token for starting the shell. IE launches a low integrity child process of itself. Certain functions, like Win32's CreateProcess check the image manifest for requestedExecutionLevel and enlist the seclogon service if needed to elevate the new process. The kernel syscall to create a process doesn't do any of this.
Windows messages are implemented by the Win32 subsystem, not the kernel (even though part of Win32 does run in kernel mode in win32k.sys). The kernel does supply several IPC methods, but not this one. Win32 does the target window integrity check (UIPI).
I would say that UAC is a user mode construct, mainly implemented in winlogon, kernel32.dll, LSA and the seclogon service. The kernel does enforce restricted tokens and integrity labels for kernel objects though. I can't say that the NT security model has been followed as well as it could be (part of this is due to its complexity), but Vista (and UAC) do use a lot more of it, and seem to have avoided serious security or compatibility problems. One issue is that NT was designed for use on a LAN with trusted programs and the security system was for protecting users and the system from unauthorized users, not from malicious programs. Thus, the owner of a computer is the Administrator, since the system doesn't have anything that user shouldn't have access to. Unfortunately, this doesn't work so well when that admin can't trust all the programs on the system to behave.
In the book, that's what happens, except that Bowman is able to get to a shelter before decompression completes.
I was thinking of one transaction, where the file is truncated and new contents are written. Either both entirely happen, or neither do.
OTOH, I think you're on to something with a simplified transaction system that allows atomic pairs of operations that can be used to implement things like this, e.g. replace one file with another, ensuring it's written completely on commit.
EFS.
/e /s on your profile dir. 5 seconds of googling surely would have revealed this.
Right click on your profile, go to properties, advanced, check the encrypt box. Alternatively, cipher
It sounds like the correct solution is for the file system to implement transactional semantics. That is what the applications need and were incidentally getting, despite it not being in the spec.
Why isn't this being considered as the solution? There are other major OSes have implemented basic atomic transactions in their filesystems successfully, why not Linux?
Then you have some other corruption problem that has nothing to do with the layout of user profiles.
Besides, I didn't have to google to determine the layout, I could see it after a minute of browsing. Google provided me with more formal documentation to reference in my post.
I never said there weren't other problems. I consider the default name for those paths to be hideous. Plus with the ridiculous and horrible limitation that is MAX_PATH in so many Win32 components, that is path real estate that we can't afford to spend. Parts of Win32 go to great lengths to work around MAX_PATH, including searching for shorter surrogate paths to use instead. .NET, foolishly built on Win32, has the same problems.
At least they changed the defaults to "Users", "Documents" and "AppData", etc in Vista.
I was responding to the specific complaint that profile paths were arcane and not understandable.
It took me 5 seconds to google some docs for user profile paths: User Data and Settings Management
Instead, the roaming stuff goes into:
C:\Documents and Settings\USERNAME\Application Data\Microsoft\Outlook
And the non-roaming stuff goes into
C:\Documents and Settings\USERNAME\Local Settings\Application Data\Microsoft\Outlook
Doesn't seem so awful.
Copy the user profile over?
What about the Windows Credentials Management subsystem? It's been there since XP. IE and Explorer use it for the remember password option. The list of usernames/passwords in your profile can be modified via the users control panel.
However, Microsoft does seem to prefer a single (or very few) signon system with an AD domain or Passport.
Can you be more specific as to Windows' "uniquely bad IO layer"?
Block allocation is the responsibility of the filesystem. Windows doesn't have a flash optimized filesystem because it would 1. break backwards and cross compatibility because MS would have to implement a new filesystem, one that they wouldn't port to previous OSes and wouldn't be compatible with other OSes, either because of NIH syndrome or because other OSes don't have a raw flash optimized FS (i.e. OSX) and 2. as the parent said most consumer hardware does not expose the raw blocks to the interface, so the FS would be of limited value.
However, there is a lot more to Windows' IO layer than filesystems, and there's nothing in the rest of the IO system preventing a raw flash optimized stack. I think Microsoft considers this to be a hardware problem, best solved in hardware, where it has been solved in hardware.
NT was written from scratch. At one point it was going to replace the old OS/2 codebase, but the breakup happened and it ended up replacing the old Windows codebase (albeit after the old line had 3 more major releases).
I was under the impression that modern Linux kernels had negligible performance impact from using a swap file as opposed to a dedicated swap partition.
Personally, I much prefer using a swap file because it gives me more flexibility in locating, resizing and moving swap.
Yeah, we just do it indirectly, through citizens, by preventing felons from voting in many states (under varying circumstances). Even for what should be petty things like possessing a few ounces of a controlled substance.
Windows NT was designed to be multi-user from its first release. Groups and privileges have always been provided for role based security.
I, for one, am glad they didn't copy a 1989 standard unixy security model with limited UGW/RWX, root or nothing privilege, primary groups, multiple object namespaces, NFS problems, etc.
Users can rearrange their own start menu and desktop without a UAC prompt or affecting the system. Users can't modify the All Users common profile that affects the start menus of all users on the system.
Explorer displays items from both locations side by side, so this isn't obvious. NT4 used to have a different icon for common items.
Many installers just dump their items into the All Users profile, which makes modifying it a pain.
Doesn't the Vista installer create a UAC administrator account that needs elevation to exercise admin privileges? Sort of like the OSX default admin account?
That's what cache hints are for. The media player should be opening the file with a hint that it will be read sequentially. On NT, this means that the cache blocks go to the end of the list, the first to be recycled.
Another problem is that the kernel doesn't know what data is important and what isn't. In Vista (and WS2008), there are 8 cache priorities, which helps fix this. Background processes run at the lowest priority and can't encroach on the high priority memory used by interactive applications.
Haskell programs can use whitespace to define blocks. However, anything that can be done with the layout rule (whitespace) can also be done with braces and semicolons, so you can have it either way.
Actually, the database backend for Exchange is the Extensible Storage Engine, previously Jet Blue. This is the same engine that Active Directory uses. ESE is an entirely different engine than Access uses, sometimes called Jet Red. ESE is just a table engine: it does not provide a high level interface like SQL. Jet Red has best-effort error recovery. ESE uses a journal.
winerror.h is also shipped with the WDK. It contains all the error codes for the Win32 environment subsystem, which isn't implemented by the kernel. Win32 exists mostly in user mode: although a sizable chunk of the Win32 server process was moved from winsrv.dll into win32k.sys in NT4, which does run in kernel mode, Win32 really isn't part of the kernel. They did that to cut down on IPC overhead, not to integrate it into lower layers. Win32 services can only be used by user mode processes.
ntstatus.h is the header that contains the kernel's error codes. That file is 16272 lines in the WDK 6000 (with 8 comment lines to every code), and contains few codes not relevant to the kernel.
It's not surprising that winerror.h contains messages not related to the kernel because it's not what I'd consider part of the kernel. AFAICT, this file is included only to support win32k.sys, and possibly display drivers. I'd be interested to know how many of the kernel source files actually include winerror.h.
Just to clarify, NtCreateProcess, the kernel syscall to create a process, is much lighter than Win32's CreateProcess function. NtCreateProcess doesn't even start any threads in the new process or load dependent libraries. CreateProcess includes out of process calls to register with CSRSS, an application compatibility database lookup since XP (so slow that there's a special option on WS2003 to turn it off), tons of registry reads, among other things.
NtCreateProcess even supports COW fork, by specifying a NULL image to load-- it's the Win32 libraries that aren't designed to support forking. POSIX^W SFU^W SUA processes are much lighter than Win32's and can use the kernel's fork support.
- An IRP is an IO Request Packet. They absolutely were used in every version of NT, including 3.1.
- A DPC is a Deferred Procedure Call-- similar to the bottom half of an interrupt handler on Linux. Those were also definitely in all versions of NT. WDM just provides a wrapper over such functions.
- At no point did NT use 16-bit code in kernel mode.
- OS2/NT was being written from scratch, but then Microsoft and IBM parted ways and it became Windows NT.
- NT 3.51 didn't introduce a new driver model. It did introduce FastIO, which is a function call oriented alternative to issuing IRPs for certain synchronous IO operations (particularly those that can be done from cache).
I can't find any surviving documentation for NT 3.1 kernel APIs, but if you do a linkLinux implements a global working set. This means that when the OS decides that it wants to remove a page from its working set (i.e. swap it to disk), it pulls that page out of all the processes that are using it (since the page might be shared), writes it to disk and marks the page as free.
Windows NT implements a per-process working set. A page is moved out of the working set of a particular process, and when it's been removed from all processes (in case it was shared), it goes into the standby cache, a sort of limbo where the page exists both in memory and disk and can't be written to (this makes it possible for the page to be moved completely to disk or back into use without further disk access). Each process has a soft minimum and maximum working set that the memory manager tries to keep a process within. Memory heavy processes have their max working set automatically expanded. Task Manager reports a process's current working set under the heading of mem usage. More pages allocated to the process may in fact be in memory in the standby list, but they won't show up in this count. Memory cached by the OS (e.g. standby cache, file cache, write cache) is not counted in the working set of any process, even if only one process is really using it. They show up as "System cache" in the performance tab, and some caches are double counted as "Available" because they can be discarded without disk access. When a process calls NtFreeVirtualMemory (the syscall for freeing private memory pages), the OS does not keep it in the process's working set. The working set is always equal or smaller than the sum of shared and private memory allocated to the process. If a process were to free all its private memory, and somehow unload all its modules and free its stack, the working set would go down to zero. A program has full control over what memory is allocated to it. It can't fully control how much of that memory is actually resident in RAM, though, and that's what is reported by Task Manager.
When you minimize a window, the Win32 subsystem sets the process's maximum working set to the system minimum, effectively moving most of its private pages into standby. Those pages will only come back into the working set as they're accessed-- it may take a long time for the working set to get has high as it was previously, and possibly never for memory leaks or unused caches. Firefox definitely implements some hefty caches.
In short: yes Windows implements a memory cache, but not for pages that have been freed.
Responding by number:
1. IO priorities. Yes, these are quite nice, although it'd be nice if more than 2 priorities were implemented (currently only background and normal are supported). Memory priorities are new in Vista as well, and are also quite useful.
3. Much of the UMDF has been backported to XP as a redistributable.
6. IO cancellation. The NT kernel has always supported cancellation of pending async IRPs. The kernel always tries to cancel pending IRPs in a dying process so it can be terminated. However, there have been places where drivers would hold on to an IRP for a long time without letting it be canceled, and some of the places where that was happening have been addressed in Vista. A notable case was during a MUP open, i.e. when the system was trying to locate the host and service behind a UNC path. Vista does now allow cancellation of a synchronous IO operation, which previously could only be done by killing the thread.
5. Shell multithreading. I don't agree that Explorer and the shell have become substantially more multithreaded. Synchronous IO is still done heavily in the GUI thread, which causes the window to stop responding. The aforementioned ability to cancel sync. IO in another thread was added to make it possible to abort such blocking, which is a poor workaround.
7. Async IO in SMB (the LanmanRedirector filesystem internally). It has always supported async IO. The new CTRL+C ability to abort is used to cancel synchronous IO in the UI thread. One issue is that some APIs can only be done synchronously, such as file opens. You can only associate a file with an IOCP after the file is opened. From the application perspective, this could be fixed by having worker threads do the blocking open operations in the background.
8. Transactions. Yes, very nice. The transaction engine is even extensible: ISVs can add their own transaction resources (MS SQL server supports this), and the DTC service can coordinate them across multiple machines.
9. VSS put to better use. Well, Microsoft has a bad habit of putting good features into the system and then dropping the ball when it comes to making use of them. It's like the teams don't communicate with each other. Something that XP did use VSS for was in ntbackup, to archive a locked file by copying the shadow version. Vista backup does not supports this, and ntbackup is incompatible with Vista.
In the end, I'm with you: until they fix the things that have been broken in Vista, I plan to stay with XP and 2003 for some time.
To be fair, Windows NT had granular, integrated and standardized security, networking, virtual memory (even in kernel mode), a unified page cache, SMP, and other advanced features right from its first release in 1993. Dave Cutler and the rest of the VMS team MS acquired definitely knew what they were doing.
The exploits that Windows has had have very rarely been kernel or core design issues. Windows has a secure design, but it's rarely configured to be secure and has suffered numerous implementation faults. In particular: usermode components, notably the shell, LSA and RPC. Microsoft is also guilty of putting compatible defaults ahead of secure ones, e.g. making Admin accounts default in XP. OEMs are also partially to blame here because they decide how the computer comes loaded from the factory (i.e. with one auto-logon admin account), network admins for allowing it to stay that way (in a corp environment), and ISVs for making tons of software that requires admin privileges when it shouldn't.
With Vista, Microsoft is trying to keep around as much old code as possible in certain components, maintain compatibility with old software, change the default privilege level for programs to non-admin, and implement some kind of TCPAesque DRM. In short, they're trying to have their cake and eat it too, via technical means. It's not pretty. Time will tell how effective it is.