Now, it would be nice if we could see what's in the cache in Windows, but you can't.
If you're really curious, you can download windbg and run !memusage from a local kernel debugging session (or using livekd). You can also use meminfo -a -f, though you'll probably need to post process the output to group used pages per file like !memusage does.
It would nice if you could peg a file to the cache, in Windows, but you can't.
Not that I would recommend that, but you can write a small app that maps a file into memory and VirtualLock's the entire view. That will make sure the file's pages remain resident.
Now you might think that after a day of builds and recompiles, at least some of that stuff would wind up in cache, but it feels like it doesn't.
It's also possible that something else makes things slow. You could profile the build process using xperf, or at least check whether these files are actually being read from disk using resmon.exe or some other disk tracing tool.
Even if it turns out that the files are in fact not cached, it's not usually the operating system's fault. Something must have pushed the cached pages out of memory. For example, maybe Visual Studio itself (or the tools it spawns, like the linker) temporarily consumes a lot of memory during the build, eating into the cache. (Though in that particular case, Superfetch should eventually reload the files back into memory)
Starting with Vista, working sets of GUI processes are no longer emptied when the main window is minimized.
For the standby cache recycle problem, Superfetch can help a lot. First of all, it can detect when apps do things like read lots of files sequentially without using FILE_FLAG_SEQUENTIAL_SCAN (or when they do this through a mapped view) and deprioritize these pages so they don't affect normal standby memory. And if useful pages still end up being recycled (e.g. because some app temporarily consumed lots of memory), Superfetch can re-populate them from disk later.
Still fails badly though, because the task manager will show lots of available memory when lot of caching is being done
And what's the problem with this? If memory is shown as available then it *is* available. When an application asks for a page of memory, and there are no free/zeroed pages left, the memory manager will take a "cached available" (a.k.a "standby") page, zero it and give it to the application.
It's not very hard to find systems that consider more than 30% memory available, but are considerably slowed down by swap activity.
It's also easy to find systems that are paging like crazy while their commit charge is nowhere near the total amount of RAM. Any workload that involves a lot of cached or memory mapped file IO can potentially cause this, because file backed pages don't consume commit but they can definitely consume a lot of RAM.
The situation you describe (lots of paging even though 30% of RAM is available) is actually not all that common. Let's say we're talking about a 2 GB system, so 30% is around 600 MB of available pages. This is far above the threshold at which the memory manager starts actively trimming working sets. That makes it pretty hard to come up with a scenario that would cause a lot of paging while maintaining at least 600 MB of available memory. Consider the steps that need to occur before a page can be hard faulted on:
A page is allocated and added to the working set of a process
The page is trimmed and placed on the modified list if it's dirty (otherwise it goes directly to standby list)
The page is written to the pagefile (or its backing file, if it's a file page) and placed on the standby list
The page is repurposed from the standby list
Finally, the page is accessed again, causing a hard fault
The first hurdle is step #2. If there is no trimming, pages will simply accumulate in process working sets (until available memory drops below the trimming threshold). And even if the app explicitly trims its pages, they will normally sit on the standby list for a long time. The standby list is FIFO, so somebody will need to push 600 MB worth of pages into it before a given page is repurposed.
Low available memory on the other hand is a great predictor of paging. If available pages get low that means the standby list is much shorter, and trimming is more frequent. Both of these are necessary conditions for paging. Also note that it doesn't matter whether RAM is being consumed by pagefile-backed or file-backed pages - available memory will drop in either case, unlike commit charge which only counts pagefile-backed pages.
If you have a system with 2 GB of memory and 3GB of commit, it's almost always a given you will soon suffer from memory starvation.
I'm typing this on a 2 GB x64 win7 machine. It's running two Office apps, Visual Studio, several browser windows and a couple of memory hogging apps that we use internally where I work. Commit charge is stable around 3.4 GB, but there is ~800 MB of available pages. Reads from the pagefile (according to resmon) are very sporadic - maybe a couple per minute. Pagefile writes are even less frequent (Page Writes/sec in perfmon shows a steady 0, and !vm 9 in the kernel debugger shows that the last write occurred 12 minutes ago).
Back in the day of Windows 2000 and XP, the Task Manager chart reported the memory comit charge. Basically, that was the amount of memory applications (and Windows) requested allocated. This does not mean that much memory was actually used, but with the exception of very badly written/buggy programs, it should be close
Not necessarily. Many programs commit large chunks of memory in case they need it later but only use a small portion initially. This simplifies program logic because you don't have to free and reallocate the buffer when you need more space, deal with potential reallocation failures etc. Or a program might want to specify a larger-than-default stack commit size to make sure it doesn't hit a stack overflow if it tries to extend the stack while the system is temporarily out of commit (most services and other system critical processes do that). Or it might map a copy-on-write view of a file, in which case commit is charged for the entire view but no extra physical memory is used until the program actually writes to the pages. Etc etc... The end result of this is that you can't really say anything conclusive about physical memory usage by looking at commit charge
Commit charge is a virtual memory metric. It's great for detecting memory leaks and deciding how big your pagefile needs to be, but not so great for understanding physical memory usage. Often it might seem like there is a correlation between commit charge and physical memory, but you can also find systems that are very low on available RAM yet have plenty of available commit, and vice versa.
Task manager now shows used physical memory (defined as Total - Available). Available memory is the most straightforward way to understand whether your system needs more memory or not, and this is why in Vista/Win7 it was chosen as the main indicator of "memory usage".
So,pray tell, where do I learn the meanings of the various stats in Task Manager?
You can press F1 while in task manager and then search for a particular metric, e.g. "available memory". This produces results that seem moderately useful, for example:
Under Physical Memory (MB), Total is the amount of RAM installed on your computer, listed in megabytes (MB). Cached refers to the amount of physical memory used recently for system resources. Available is the amount of memory that's immediately available for use by processes, drivers, or the operating system. Free is the amount of memory that is currently unused or doesn't contain useful information (unlike cached files, which do contain useful information).
For more details about particular counters you can check the Windows Internals book, or Memory Performance Information on MSDN. Also, many counters in task manager have similar or identical perfmon counters, and perfmon has its own help (IIRC there's a "show description" option in the counter selection dialog)
> The Win7 task manager does show a "cached" stat, though, so your effectively free memory is "free"+"cached".
Not quite. "Cached" includes pages on the modified list, which are not immediately available (or may not be available at all, if your pagefile is full/disabled... (which is one of the reasons why you shouldn't disable it, BTW))
"Effectively free" memory is shown as "Available" in win7's task manager. It is the sum of "Free" (or more precisely, free+zeroed) and "Standby". Check out the memory usage chart in the the resource monitor to get a better idea of how all these counters relate to each other.
I don't think this does anything to undercut my basic assertion that Win 7 is really a service pack for Vista
Windows service packs are mostly collections of bug fixes. New features or large under-the-hood changes in service packs are extremely rare (things like the new firewall in XP SP2 are an exception rather than the rule). Something like a completely different taskbar UI would never make it into a service pack, not to mention things like a major rewrite of the scheduler, or extensive changes in the memory manager (e.g. the removal of the PFN lock).
Yet in May of 2007 Slashdot reported that Microsoft announced that Vista was to be it's last 32 bit OS and that the sucessor to Vista would be 64 bit only. See here...
Actually, what was announced was that WS 2008 would be the last 32-bit *server* release. This is even mentioned in some of the comments from the link you supplied:
Win7 on 512 MB works fine for simple tasks like browsing.
> That says to me that the 700MB commit isn't all cache
Cached file data is not even included in the commit charge.
You can't actually make any conclusions regarding physical memory usage based on the 700 MB number. Part of that 700 MB is resident in RAM, another part is in the paging file, and yet another is purely virtual - it doesn't exist anywhere until the corresponding pages are accessed by the application (think guard pages in thread stacks etc). Commit charge tells you how much *virtual* memory is in use. For RAM usage, check out the "physical memory usage" graph in task manager.
there is no way that Vista-64 or Win7-64 will gift 32-bit apps with more than the 2GB of address space that they are currently allowed (vs. the full 4GB that OS X gives to 32-bit apps)
For apps that can handle addresses above 2 GB, 64 bit Vista (and XP) already does exactly that:
Then use "end process", not "end task". End Task sends a message to the target window then waits 5 seconds or so for the app to exit gracefully. End Process kills the process right away.
Starting with XP, the default permissions on the root directory are such that regular users can't create files there. For Win2k there's a KB article describing how to set permissions to prevent a similar kind of attack:
Process\Working Set and Process\Private Bytes (which is what VM Size really is) are both not very good indicators of physical memory usage, but Working Set is definitely closer. Even better is doing what you mentioned - flush working set by calling EmptyWorkingSet or minimizing the app, then perform some typical tasks and measure the working set again. This will give you the number of pages touched since the working set was flushed, which is a reasonably good definition of "RAM usage".
The reason why working set can be larger than private bytes is not because it includes some kind of "free" memory that the OS can just grab and reuse for other purposes; it's mostly because private bytes doesn't include MEM_MAPPED or MEM_IMAGE pages both of which can be part of working set.
MS made a mindbogglingly stupid choice when they made sizeof(long) = 32bits in their 64 bit data model. Every other 64 bit operating system made sizeof(long) == 64 bits. That means that even if you've ported to 64 bits before (because you're a server app that works on thing other than Windows), you're up for porting work.
So you had existing code which wasn't 64 bit clean. And you "fixed" it by assuming that on any 64 bit system sizeof(long) == sizeof(void*)? I'd say you deserve what you get if that's the case.
You should have used proper types (size_t, ptrdiff_t, or an explicit 64 bit integer type) to begin with, and if you're still not using them even after you've "ported" the code the first time then you have only yourself to blame.
Also,.NET apps which thunk to external 32-bit DLL's for added functionality won't work with the 64-bit.NET runtime (e.g. - if you call out to kernel32.dll or any of the standard Win32 DLL's your code will, of course, not work with 64-bit DLL's).
As long as you use IntPtr for pointer types it will work just fine. 64 bit apps will use the 64 bit version of kernel32.dll.
Of course it's always possible to introduce bugs if you try to be too clever, e.g.:
Is ActiveX a bad idea: Yes. Implemented poorly: YES
Do you mean ActiveX in general? Or ActiveX support in IE?
Saying that ActiveX in general is a bad idea is like saying DLLs are bad. An ActiveX object is nothing more than a component packaged in a DLL that can be used in a somewhat language-neutral way.
ActiveX support in IE could have been better, I'll grant you that.
Is WSH a bad idea: No. Implemented poorly: Yes. (If not, it would not be abused. Since it has to be turned off to secure a system...it's not very valuable. Other scripting languages on other systems don't have these problems.)
I don't understand this. Can you safely run a Perl script from an untrusted source? Do you have to disable Perl in order to secure a system? Why do you think the answers to these questions are different when you replace Perl with WSH?
Microsoft has a problem and instead of fixing it they reluctantly disable the defective part and allow it to be turned right back on again.
It's not like there's a checkbox in Outlook saying "allow.vbs attachments". To do this you'd have to edit the registry. Seems pretty reasonable to me.
If WSH and other executibles are turned off by default in Outlook or at the system level, why have them at all?
I'm not sure what you're asking here. How do you imagine using a computer if you can't execute any executables?
Having WSH and other disabled dangerous services on the system at all makes turning them right back on again that much easier.
So what you're saying is that since.vbs files can be abused, they should be permanently disabled? By this logic, you'd have to disable.exe and.bat files as well.
Slashdot Mirror
If you're really curious, you can download windbg and run !memusage from a local kernel debugging session (or using livekd). You can also use meminfo -a -f, though you'll probably need to post process the output to group used pages per file like !memusage does.
Not that I would recommend that, but you can write a small app that maps a file into memory and VirtualLock's the entire view. That will make sure the file's pages remain resident.
It's also possible that something else makes things slow. You could profile the build process using xperf, or at least check whether these files are actually being read from disk using resmon.exe or some other disk tracing tool.
Even if it turns out that the files are in fact not cached, it's not usually the operating system's fault. Something must have pushed the cached pages out of memory. For example, maybe Visual Studio itself (or the tools it spawns, like the linker) temporarily consumes a lot of memory during the build, eating into the cache. (Though in that particular case, Superfetch should eventually reload the files back into memory)
Turning off the pagefile will not prevent DLL/EXE pages from being paged out, which is what the the person you're replying to complained about.
Starting with Vista, working sets of GUI processes are no longer emptied when the main window is minimized.
For the standby cache recycle problem, Superfetch can help a lot. First of all, it can detect when apps do things like read lots of files sequentially without using FILE_FLAG_SEQUENTIAL_SCAN (or when they do this through a mapped view) and deprioritize these pages so they don't affect normal standby memory. And if useful pages still end up being recycled (e.g. because some app temporarily consumed lots of memory), Superfetch can re-populate them from disk later.
And what's the problem with this? If memory is shown as available then it *is* available. When an application asks for a page of memory, and there are no free/zeroed pages left, the memory manager will take a "cached available" (a.k.a "standby") page, zero it and give it to the application.
It's also easy to find systems that are paging like crazy while their commit charge is nowhere near the total amount of RAM. Any workload that involves a lot of cached or memory mapped file IO can potentially cause this, because file backed pages don't consume commit but they can definitely consume a lot of RAM.
The situation you describe (lots of paging even though 30% of RAM is available) is actually not all that common. Let's say we're talking about a 2 GB system, so 30% is around 600 MB of available pages. This is far above the threshold at which the memory manager starts actively trimming working sets. That makes it pretty hard to come up with a scenario that would cause a lot of paging while maintaining at least 600 MB of available memory. Consider the steps that need to occur before a page can be hard faulted on:
The first hurdle is step #2. If there is no trimming, pages will simply accumulate in process working sets (until available memory drops below the trimming threshold). And even if the app explicitly trims its pages, they will normally sit on the standby list for a long time. The standby list is FIFO, so somebody will need to push 600 MB worth of pages into it before a given page is repurposed.
Low available memory on the other hand is a great predictor of paging. If available pages get low that means the standby list is much shorter, and trimming is more frequent. Both of these are necessary conditions for paging. Also note that it doesn't matter whether RAM is being consumed by pagefile-backed or file-backed pages - available memory will drop in either case, unlike commit charge which only counts pagefile-backed pages.
I'm typing this on a 2 GB x64 win7 machine. It's running two Office apps, Visual Studio, several browser windows and a couple of memory hogging apps that we use internally where I work. Commit charge is stable around 3.4 GB, but there is ~800 MB of available pages. Reads from the pagefile (according to resmon) are very sporadic - maybe a couple per minute. Pagefile writes are even less frequent (Page Writes/sec in perfmon shows a steady 0, and !vm 9 in the kernel debugger shows that the last write occurred 12 minutes ago).
Not necessarily. Many programs commit large chunks of memory in case they need it later but only use a small portion initially. This simplifies program logic because you don't have to free and reallocate the buffer when you need more space, deal with potential reallocation failures etc. Or a program might want to specify a larger-than-default stack commit size to make sure it doesn't hit a stack overflow if it tries to extend the stack while the system is temporarily out of commit (most services and other system critical processes do that). Or it might map a copy-on-write view of a file, in which case commit is charged for the entire view but no extra physical memory is used until the program actually writes to the pages. Etc etc... The end result of this is that you can't really say anything conclusive about physical memory usage by looking at commit charge
Commit charge is a virtual memory metric. It's great for detecting memory leaks and deciding how big your pagefile needs to be, but not so great for understanding physical memory usage. Often it might seem like there is a correlation between commit charge and physical memory, but you can also find systems that are very low on available RAM yet have plenty of available commit, and vice versa.
Task manager now shows used physical memory (defined as Total - Available). Available memory is the most straightforward way to understand whether your system needs more memory or not, and this is why in Vista/Win7 it was chosen as the main indicator of "memory usage".
You can press F1 while in task manager and then search for a particular metric, e.g. "available memory". This produces results that seem moderately useful, for example:
Under Physical Memory (MB), Total is the amount of RAM installed on your computer, listed in megabytes (MB). Cached refers to the amount of physical memory used recently for system resources. Available is the amount of memory that's immediately available for use by processes, drivers, or the operating system. Free is the amount of memory that is currently unused or doesn't contain useful information (unlike cached files, which do contain useful information).
For more details about particular counters you can check the Windows Internals book, or Memory Performance Information on MSDN. Also, many counters in task manager have similar or identical perfmon counters, and perfmon has its own help (IIRC there's a "show description" option in the counter selection dialog)
.
> The Win7 task manager does show a "cached" stat, though, so your effectively free memory is "free"+"cached".
Not quite. "Cached" includes pages on the modified list, which are not immediately available (or may not be available at all, if your pagefile is full/disabled... (which is one of the reasons why you shouldn't disable it, BTW))
"Effectively free" memory is shown as "Available" in win7's task manager. It is the sum of "Free" (or more precisely, free+zeroed) and "Standby". Check out the memory usage chart in the the resource monitor to get a better idea of how all these counters relate to each other.
I don't think this does anything to undercut my basic assertion that Win 7 is really a service pack for Vista
Windows service packs are mostly collections of bug fixes. New features or large under-the-hood changes in service packs are extremely rare (things like the new firewall in XP SP2 are an exception rather than the rule). Something like a completely different taskbar UI would never make it into a service pack, not to mention things like a major rewrite of the scheduler, or extensive changes in the memory manager (e.g. the removal of the PFN lock).
Yet in May of 2007 Slashdot reported that Microsoft announced that Vista was to be it's last 32 bit OS and that the sucessor to Vista would be 64 bit only. See here...
Actually, what was announced was that WS 2008 would be the last 32-bit *server* release. This is even mentioned in some of the comments from the link you supplied:
http://slashdot.org/comments.pl?sid=235071&cid=19172261
Which is exactly what happened by the way - the server edition of win7 (WS 2008 R2) is 64-bit only.
Win7 on 512 MB works fine for simple tasks like browsing.
> That says to me that the 700MB commit isn't all cache
Cached file data is not even included in the commit charge.
You can't actually make any conclusions regarding physical memory usage based on the 700 MB number. Part of that 700 MB is resident in RAM, another part is in the paging file, and yet another is purely virtual - it doesn't exist anywhere until the corresponding pages are accessed by the application (think guard pages in thread stacks etc). Commit charge tells you how much *virtual* memory is in use. For RAM usage, check out the "physical memory usage" graph in task manager.
On vista you can do the same thing using bcdedit:
bcdedit /set badmemorylist 0x12345 0x23456
Parameters are page frame numbers.
For apps that can handle addresses above 2 GB, 64 bit Vista (and XP) already does exactly that:
Memory and Address Space Limits
Actually, what was said is that WS08 would be the last 32-bit *server* OS. Which it is (32-bit win7 is desktop-only).
Then use "end process", not "end task". End Task sends a message to the target window then waits 5 seconds or so for the app to exit gracefully. End Process kills the process right away.
http://support.microsoft.com/default.aspx?scid=kb; en-us;327522
The reason why working set can be larger than private bytes is not because it includes some kind of "free" memory that the OS can just grab and reuse for other purposes; it's mostly because private bytes doesn't include MEM_MAPPED or MEM_IMAGE pages both of which can be part of working set.
So you had existing code which wasn't 64 bit clean. And you "fixed" it by assuming that on any 64 bit system sizeof(long) == sizeof(void*)? I'd say you deserve what you get if that's the case.
You should have used proper types (size_t, ptrdiff_t, or an explicit 64 bit integer type) to begin with, and if you're still not using them even after you've "ported" the code the first time then you have only yourself to blame.
For now, it is only known to work on the Virtutech simulator, since no x86-64 hardware is available yet.
x64 XP was developed the same way:
This basically means the port was already done on an emulator by the time the actual hardware arrived.
As long as you use IntPtr for pointer types it will work just fine. 64 bit apps will use the 64 bit version of kernel32.dll.
Of course it's always possible to introduce bugs if you try to be too clever, e.g.:
http://blogs.msdn.com/joshwil/archive/2004/03/16/9 0612.aspx
But reasonably written code should work on 64 bit without any modifications.
I think you mean "sarcasm".
How ironic that you don't know the meaning of ironic.
So you're saying that his post *did* contain irony after all, right?
Reg.exe is shipped as part of the OS starting with XP.
Things like FileMon, RegMon, Process Explorer etc.
Actually the dialog says: "Windows Firewall has blocked this program from accepting connections from the Internet".
Note that it's "accepting", not "initiating".
This is why your browser can make HTTP connections even though it's not on the exceptions list.
Do you mean ActiveX in general? Or ActiveX support in IE?
Saying that ActiveX in general is a bad idea is like saying DLLs are bad. An ActiveX object is nothing more than a component packaged in a DLL that can be used in a somewhat language-neutral way.
ActiveX support in IE could have been better, I'll grant you that.
Is WSH a bad idea: No. Implemented poorly: Yes. (If not, it would not be abused. Since it has to be turned off to secure a system...it's not very valuable. Other scripting languages on other systems don't have these problems.)
I don't understand this. Can you safely run a Perl script from an untrusted source? Do you have to disable Perl in order to secure a system? Why do you think the answers to these questions are different when you replace Perl with WSH?
It's not like there's a checkbox in Outlook saying "allow .vbs attachments". To do this you'd have to edit the registry. Seems pretty reasonable to me.
If WSH and other executibles are turned off by default in Outlook or at the system level, why have them at all?
I'm not sure what you're asking here. How do you imagine using a computer if you can't execute any executables?
Having WSH and other disabled dangerous services on the system at all makes turning them right back on again that much easier.
So what you're saying is that since .vbs files can be abused, they should be permanently disabled? By this logic, you'd have to disable .exe and .bat files as well.