Ars Analysis Calls Windows 7 Memory Usage Claims "Scaremongering"

Via newsycombinator comes a reaction at Ars Technica to the recently reported claims of excessive memory use on machines running Windows 7. From the article: "I installed the XPnet performance monitoring tool and waited for it to upload my data to see what it might be complaining about. The cause of the problem was immediately apparent. It's no secret that Windows 7, just like Windows Vista before it, includes aggressive disk caching. The SuperFetch technology causes Windows to preload certain data if the OS detects that it is used regularly, even if there is no specific need for it at any given moment. Though SuperFetch is a little less aggressive in Windows 7, it will still use a substantial amount of memory—but with an important proviso. The OS will only use memory for cache when there is no other demand for that memory."

Comments

So

[sopssa]

Though SuperFetch is a little less aggressive in Windows 7, it will still use a substantial amount of memory—but with an important proviso. The OS will only use memory for cache when there is no other demand for that memory.

I really wonder when people will get this. In the earlier thread I saw people commenting that Windows 95 didn't need so much memory and so on..

To state it again. This is not RAM memory you need, use or have purpose for. IF you do need it, it is zeroed-out and free'd to application in like 30ms (one frame in usual FPS games).

If you have fast memory, do use it to it's full extend.

Re:So

[ls671]

Yep, same for Linux. My Linux boxes use ALL the memory available even if I do not run many applications on it. The left over memory SHOULD be used as buffers/cache. If Windows 7 seems to use more memory from a newbie point of view, it might be because it does things like it should better than previous versions. I can't tell for sure since I have never tried win 7.

See this 4 GB Linux machine below, it only has ~49 MB of "absolutely free" memory and uses ~449 MB of swap.

In realty, it has ~2842 MB of "available memory" since it uses ~2792 MB of buffer/cache.

Using buffer/cache makes the system order of magnitude faster. If programs need that memory, the OS will give to them and use less buffer/cache.

free
                          total used free shared buffers cached
Mem: 4133252 4083380 49872 0 26852 2766248
-/+ buffers/cache: 1290280 2842972
Swap: 1999800 449244 1550556

Re:So

[Rockoon]

You're as bad as the guys who wrote that article in the first place. If you don't know how Windows works, please don't talk about it.

Hell, its not just windows. All operating systems do this.. and to be quite frank, programmers of all kinds should have cache techniques well understood. So the GP is neither a windows guru nor a decent programmer. The odds are very good that hes just an I-use-software geek, rather than someone who knows anything about computers.

Re:So

[berboot]

SuperFetch also keeps track of what times of day that applications are used, which allows it to intelligently pre-load information that is expected to be used in the near future.
Source: Wikipedia
Their work, says Horvitz, was able to predict which applications users would open by time of day and also by day of the week.
Source: InfoWeek

Re:So

[m_pll]

So,pray tell, where do I learn the meanings of the various stats in Task Manager?

You can press F1 while in task manager and then search for a particular metric, e.g. "available memory". This produces results that seem moderately useful, for example:

Under Physical Memory (MB), Total is the amount of RAM installed on your computer, listed in megabytes (MB). Cached refers to the amount of physical memory used recently for system resources. Available is the amount of memory that's immediately available for use by processes, drivers, or the operating system. Free is the amount of memory that is currently unused or doesn't contain useful information (unlike cached files, which do contain useful information).

For more details about particular counters you can check the Windows Internals book, or Memory Performance Information on MSDN. Also, many counters in task manager have similar or identical perfmon counters, and perfmon has its own help (IIRC there's a "show description" option in the counter selection dialog)

.

Re:So

[Foolhardy]

I once went looking to see if there was a way to do it from within the application code itself - something like mlock()/mlockall() in posix - and I couldn't find an equivalent, which may just be a reflection of my own inexperience with the Windows API but I figured I would throw that out there anyway.

The function you're looking for is VirtualLock. You may also look into increasing the process's minimum working set with SetProcessWorkingSetSize. This requires SeIncreaseBasePriorityPrivilege.

A process that is scanning through a file is supposed to use the FILE_FLAG_SEQUENTIAL_SCAN hint so that the cached pages are recycled first, but that doesn't always happen. It also doesn't help that csrss will ask the kernel to minimize a process's working set when its main window is minimized.

Re:So

[m_pll]

Starting with Vista, working sets of GUI processes are no longer emptied when the main window is minimized.

For the standby cache recycle problem, Superfetch can help a lot. First of all, it can detect when apps do things like read lots of files sequentially without using FILE_FLAG_SEQUENTIAL_SCAN (or when they do this through a mapped view) and deprioritize these pages so they don't affect normal standby memory. And if useful pages still end up being recycled (e.g. because some app temporarily consumed lots of memory), Superfetch can re-populate them from disk later.

Linux does that

[Animats]

Linux uses available memory for cache, and rather aggressively. All available memory can be filled with cached file blocks. This happens routinely on systems which have big randomly-accessed files open, like databases.

There's nothing wrong with this, except that, once in a while, Linux hits a race condition in prune_one_dentry, causing an "oops" crash, when there's an unblockable need for a memory page and something is locking the file block cache.

This is one of the Great Unsolved Mysteries of Linux. Linus wrote about it in 2001 ("I'll try to think about it some more, but I'd love to have more reports to go on to try to find a pattern.. "). As of 2009, this area is still giving trouble. The locking in this area is very complex.

Re:30ms?

[ashridah]

You might want to grab a copy of Process Explorer sometime, and look at the stats it reports. you'll notice that windows actually spends idle time pre-zeroing ram, so that this is already done, in more than enough amounts. If your system is slammed, i could see having to pre-zero the pages, just before use, however, but it's not like it's not something that couldn't be done while waiting for other I/O operations to complete (since your system is slammed anyway :) )

My laptop currently has 2.8 million pages zeroed atm (it has 8gb, and I don't have much running right now, so there's not a lot to cache.)

that was supposition

[YesIAmAScript]

The only data in the article you refer to was captured data from XPnet that said that >90% of RAM was in use in Windows 7 machines. There was no data saying it led to swapping, that was supposition.

http://www.computerworld.com/s/article/9158258/Most_Windows_7_PCs_max_out_memory

Re:Windows 7 and Vista Lie about memory usage

[m_pll]

Back in the day of Windows 2000 and XP, the Task Manager chart reported the memory comit charge. Basically, that was the amount of memory applications (and Windows) requested allocated. This does not mean that much memory was actually used, but with the exception of very badly written/buggy programs, it should be close

Not necessarily. Many programs commit large chunks of memory in case they need it later but only use a small portion initially. This simplifies program logic because you don't have to free and reallocate the buffer when you need more space, deal with potential reallocation failures etc. Or a program might want to specify a larger-than-default stack commit size to make sure it doesn't hit a stack overflow if it tries to extend the stack while the system is temporarily out of commit (most services and other system critical processes do that). Or it might map a copy-on-write view of a file, in which case commit is charged for the entire view but no extra physical memory is used until the program actually writes to the pages. Etc etc... The end result of this is that you can't really say anything conclusive about physical memory usage by looking at commit charge

Commit charge is a virtual memory metric. It's great for detecting memory leaks and deciding how big your pagefile needs to be, but not so great for understanding physical memory usage. Often it might seem like there is a correlation between commit charge and physical memory, but you can also find systems that are very low on available RAM yet have plenty of available commit, and vice versa.

Task manager now shows used physical memory (defined as Total - Available). Available memory is the most straightforward way to understand whether your system needs more memory or not, and this is why in Vista/Win7 it was chosen as the main indicator of "memory usage".