86% of Windows 7 PCs Maxing Out Memory

CWmike writes "Citing data from Devil Mountain Software's community-based Exo.performance.network (XPnet), Craig Barth, the company's chief technology officer, said that new metrics reveal an unsettling trend. On average, 86% of Windows 7 machines in the XPnet pool are regularly consuming 90%-95% of their available RAM, resulting in slow-downs as the systems were forced to increasingly turn to disk-based virtual memory to handle tasks. The 86% mark for Windows 7 is more than twice the average number of Windows XP machines that run at the memory 'saturation' point, and this comes despite more RAM being available on most Windows 7 machines. 'This is alarming,' Barth said of Windows 7 machines' resource consumption. 'For the OS to be pushing the hardware limits this quickly is amazing. Windows 7 is not the lean, mean version of Vista that you may think it is.'"

Comments

When do people get this

[sopssa]

RAM is wasted when it isn't in use. The fact that the task manager in Windows says your RAM is used 95% tells nothing, and no it won't "result in slow-downs as the systems were forced to increasingly turn to disk-based virtual memory to handle tasks". I'm actually really surprised, and not in a good way, that "chief technology officer" of the company doesn't know this.

The new memory models in recent OS's try to utilize all the available RAM (as they should) to speed up things otherwise. It makes a lot of sense to cache things from hard-drive in low-peak usage points, and in such such way that it doesn't interfere with other perfomance. When the things that are most often used are already cached in RAM, their loading works a lot faster. This doesn't include only files, icons or such, but everything the OS could use or do that takes time.

If theres a sudden need for more RAM, the cached data can be "dropped" in no time. It doesn't matter if it averages at 25% or 95%, just that the perfomance overally is better when you utilize all the resources you can to speed up things in general.

Re:When do people get this

[Mr Thinly Sliced]

My understanding was that memory used for disk caching doesn't show up in task manager as "used".

It's been a while since I booted win7 though, so I might be mistaken.

Certainly under linux ram used as disk cache is marked "free".

It wouldn't surprise me that win7 has a heavier memory footprint though - as more applications move to .net and web browsers use lots of flash / silverlight etc - all of these things have a RAM cost.

Re:When do people get this

[Anonymous Coward]

I think the issue here is that the system is turning to swap. Caching stuff that may be referenced again is fine and dandy, but if the system regularly turns to swap just to keep itself afloat, then you have a problem.

Re:When do people get this

[jibjibjib]

Using more RAM doesn't use more energy. Either your RAM is powered on, or it's not. And if it's powered on it maintains its contents, no matter whether the OS has actually written anything useful to it.

Re:When do people get this

[Sockatume]

If they'd measured pagefaults, they could've reported pagefaults. They didn't. RAM usage appears to be the total basis for the article, so his concern is a genuine one. We don't know enough about the study at this stage to dismiss it.

Re:When do people get this

[snemarch]

It shows up as part of the memory commit bar - which is what regular users will look at, and then go off screaming about "OMG IT USES ALL MY SYSTEM MEMORY!1!!! one one". It's also deducted from the "free" count, since technically it isn't free (it can be freed quickly, but has to be zeroed out before it can be handed off to a new app - security and all).

The Win7 task manager does show a "cached" stat, though, so your effectively free memory is "free"+"cached". And if you want more comprehensive memory stats, you should look at perfmon.msc or SysInternals' Process Explorer.

I wonder if TFA has actually measured that disk swapping happens (easy with procexp or perfmon), or are just shouting their heads off without understanding what's going on... it's well-known that SuperFetch utilizes otherwise unused memory for disk caching, and does so proactively :)

Re:When do people get this

[jimicus]

Here we see why /. needs a "-1, Wrong" mod.

Re:When do people get this

[phatcabbage]

Barth acknowledged that XPnet's data couldn't determine whether the memory usage was by the operating system itself, or an increased number of applications.
So yeah, it doesn't seem like the author really knows what's going on...

Re:When do people get this

[dhavleak]

If the memory was freed up dynamically as needed then no processes would ever be forced to resort to disk-based virtual memory.

The trouble is, the TFA doesn't actually say (at least not clearly) that the Win7 machines are indeed turning to swap more regularly. It just states that fetching stuff from the swap file is a consequence of running out of RAM and causes perf degradation. So if the Win7 machines are indeed utilizing all available RAM and yet not swapping at a significanly higher rate, it means they're making more optimum use of available RAM.

Re:When do people get this

[nmg196]

Totally agree. If you don't want Windows 7 to use the 4GB of RAM you've paid for to speed up your computer, take out 2GB and put it in the drawer. Otherwise, be thankful that it's actually making the most of the RAM you're using.

What next? People complaining that games use 100% CPU to give them maximum framerate when it could just use 30% CPU and give them 10 FPS?

Re:When do people get this

[snemarch]

Yep, that would be a problem - but neither the TFA nor xpnet mentions if this is actually happening, it seems that they're looking almost exclusively at "free physical memory", which isn't a useful stat in this regard. The xpnet site does say they factor in "how often it relies on virtual memory", but not how they do this (there's multiple metrics to choose from, some fairly uninteresting) and the fact that they seem to factor this in as a part of "memory usage" rather than keeping it as a separate stat makes me pretty wary of trusting any analysis from them.

Re:When do people get this

[lcarnevale]

HDD caching and swap are two completely different things. HDD caching is loading things from the disk to RAM to speed up things. SWAP is using the HDD as extra RAM when the system doesn't have any more memory left to use. So, what I think you wanted to do was to turn off swap, not hdd caching.

Re:When do people get this

[FooBarWidget]

Definitely the part about HDD caching slowing things down. Even in the DOS age it was well known that hdd caching utilities (I forgot the names, too long ago) improve disk performance tremendously.

Linux does the same things as Windows: it caches as much stuff from disk into main memory as possible. Try running:

    cat large_video_file.avi > /dev/null

You'll see that after running the command, your memory usage jumps up by the size of the video file. Now try running the same command again, it's now an order of a magnitude faster.

On Linux things like this are stored in main memory in the form of caches and buffers. I don't know about Windows, but Linux clears some caches and buffers if applications need real memory. Caches and buffers show up in memory usage reporting tools like 'free', so it's quite normal to see Linux systems using 90% or more RAM, most of which go to caches and buffers. It seems that most people who complain about memory usage don't know how memory is managed on modern operating systems, so they go all apeshit about "OMG HELP linux is using so much memory it sux0rz!!!" and I have to explain again and again how they're not getting it. Same goes to you. Now, Windows is suffering from the same problem.

FYI, here's the memory usage of my Linux server:

             total       used       free     shared    buffers     cached
Mem:           720        702         17          0         55        510
-/+ buffers/cache:        136        583
Swap:          399          0        399

It says 702 MB of used memory. Now look at "-/+ buffers/cache", it says 136 MB. That's the amount of memory *actually* used by applications.

Re:When do people get this

[cgenman]

Read TFA. It just claims that Windows 7 consumes all available RAM. That is the "empirical evidence." System slowdown was NOT measured.

Utilizing all available RAM is a pretty well understood technique at this point. All web browsers do this now, as do many other applications. One would expect a well-designed modern OS to do this. Consuming all memory itself is not a sign of poor programming itself, so long as disk caching of things that should be in RAM doesn't occur. This is not something that the people in the article has measured.

I'm going to side with the guy who appears to make his living testing these sorts of things.

Bad science is bad irrespective of the person conducting it. And whatever the original tester said is getting filtered through the viewpoint of the gentleman writing the article. Considering that he says that "Windows 7 is not the lean, mean version of Vista that you may think it is," yet never once compares statistics to Vista (or even mentions it outside of this statement), I'd take the conclusions from these stats with a grain of salt.

Re:When do people get this

[Moryath]

You obviously don't understand memory access design. It's all about feeding the CPU. There are two sorts of relationships we can use to make this work: temporal and sequential.

Hard drives are the largest-capacity storage (well unless you want to go to tape). But they're slow. Even the fastest high-RPM SCSI or SATA drives are SLOW compared to what's above them. This is mitigated, somewhat, by putting some cache memory on the drive's controller board itself. Still, having to "hit" the hard drive for information is, as you say, a slowdown. Same goes for "external" storage (Optical media, USB media, etc).

So you try to keep as much information as possible in RAM (next step up). Hitting RAM is less expensive than hitting the H/D in terms of a performance hit. In the original days of computing (up until the 486DX line for Intel CPUs), RAM and CPU operated on a 1:1 clock speed match, so that was that.

Once you factor in the "clock multiplier" of later CPU's, even the fastest RAM available today can't keep from "starving" the CPU. So we add in cache - L3, L2, and L1. the 486 implemented 8KB (yeah a whole 8K, wow!) in order to keep itself from starving. L3 is the "slowest", but largest, L2 is faster still but smaller, and L1's the smallest of all, but the fastest because it is literally on the same die as the CPU. That distinction is important, and in general you'll find that a "slower" CPU with more L1 Cache will benchmark better than a "faster" CPU with less.

The CPU looks for what it wants as follows:
- I want something. Is it in L1? Nope.
- Is it in L2? Nope.
- Is it in L3? Nope.
- Is it in RAM? Nope.
- Is it in the H/D Cache? (helps avoid spin-up and seek times) Nope.
- Crap, it's on the H/D. Big performance hit.

Everything except for the L1 check, technically, was a performance it. The reason for pre-caching things (based on temporal and sequential relationships) is all about predicting and getting what will be needed next into the fastest available place.

Yes, I suppose you can run an entire system where it all goes into "RAM", and you'll see it as "more responsive" simply because you never have to touch the hard drive. But turning off HDD caching is a BAD idea. It makes cache misses that much more expensive because then, instead of having even the chance of finding what you needed in RAM or in the HD's onboard cache, you have to wait for the H/D to spin up and seek to the right sector.

Re:When do people get this

[snemarch]

actaully the windows 7 caching model is great. on games the difference between the first loading of a level and subsequent loads are night and day thanks to it's caching model.

That's the windows cache system generally, from way back in the NT days... Vista and later SuperFetch is more than that.

btw, regarding the article more directly: they shows no figure about the actual _swap_ usage, a thing that may or may not disprove their theory.

Indeed. The xpnet site does mention that they factor in paging somehow, but that's still pretty useless - paging activity needs to be a separate statistic. Also, simply looking at pagefile usage isn't terribly useful, an inactive app can have it's working set trimmed and pages flushed out to disk, and this won't matter much in the big picture.

What you need to look at is the rate of pagefile activity (ie., pages/second) as well as how often it happens - not just static numbers (even if having 1gig of data in the pf is probably a warning sign :))

Re:When do people get this

[afidel]

Actually what you really need to do is calculate hard pages/second which for some retarded reason isn't available as a default stat counter, ie those pages which actually go to the secondary backing store (disk).

Re:When do people get this

[afidel]

Windows gets really cranky when it doesn't have a pagefile. We tried it for performance reasons and we saw an almost 40% drop in performance despite the server not being under any kind of memory pressure.

Re:When do people get this

[TheLink]

Yeah I don't know why they don't set up the counter by default.

Anyway to set it up yourself:

Start perfmon.msc
Then add counters
go to Memory, add "Pages Output/sec".

I'm not an authority on virtual memory but from what I know:
Page Faults/sec is not usually relevant for this - the virtual memory stuff will have page fault even if it's not swapping to/from disk - it's part of how virtual memory works.
Page Inputs/sec could happen when you launch programs (then the O/S starts paging in the stuff it needs) - it's no indication of running out of memory.
Page Output/sec on the other hand is when the O/S is low and needs to copy stuff in RAM and write it OUT to disk so that it can reuse that RAM for something else. This is the one you want to monitor.

Re:When do people get this

[xigxag]

You missed the most important emphasis:

delays in I/O processing - ostensibly due to heavy virtual memory activity as Windows compensates for insufficient RAM. (emphasis mine)

Ostensibly means "to all outward appearances." In other words, they're admitting they don't really know the inner workings, the true cause of the delays. They're just supposing it's due to RAM swapping, as opposed to increased networking activity, aero glass, more concurrent programs being run on average, or any other number of other wag'd reasons. Basically, they picked two measurements that are both higher in W7, and just said, "Well, it stands to reason that A causes B." What's that phrase that internet smarty-pantses use all the time about this?

Re:When do people get this

[Daengbo]

You mean your machine with 8GB RAM never hits swap? Wow. Shock. Color me surprised! O_o

It's called SuperFetch

[dhavleak]

I guess Devil Mountain or whoever don't know about SuperFetch. Or need publicity.

And I guess slashdot editors don't know about SuperFetch. Or maybe an article like this gets them more traffic, revenue, etc.

The fucking bullshit that passes for articles these days..

Trollworthy rating for posting this...

[hitech69]

Computerworld should just close up shop for this worthless piece of journalism, or at least give their author the boot for doing any work with Craig Barth who represents a team of morons. samzenpus should be given a troll rating for getting this to Slashdot.

Re:Depends on what kind of memory

[A beautiful mind]

It's Windows. It might be a bug.

Page Faults

[Dr_Barnowl]

The metric to count is the number of page faults, an indicator of the number of times that the OS addresses memory that isn't in RAM.

As others point out, measuring just the fraction of memory consumption is stupid. I have 6GB of RAM ; my processes are using about 1.7GB of that, but the OS is claiming that 3.8GB is consumed. So that's 2.1GB of cached data that I no longer have to wait for from disk. Hooray.

TFA hints that they may be measuring page faults, and does mention that Win7 is hitting the disk for virtual memory more often. But they should make that clearer if it's the case.

Oh come on

[megla]

First we had submitters who didn't read the stories they were posting. Then we had editors who didn't read the stories they were approving. Now we have companies who don't read the articles they put out. Seriously, it's called a file cache. That's how it's supposed to work. Nice job, idiots.

Parent is +1 informative

[Anonymous Coward]

You cannot study virtual memory performance without considering how many page faults occur.

It is perfectly reasonable to use RAM as a filesystem cache, which is why Linux has adopted this approach. The effect is that almost all of the physical RAM is always in use. The cost is that pages are more likely to be wrongly swapped out - however, in typical cases, this increased cost is tiny in relation to the huge reduction in the number of disk accesses.

Available memory != Free memory

[TheLink]

Yeah. I don't have low mem problems with Windows 7. There's stuff I don't like about Windows 7 but "memory hog" is not on the list.

For work I'm using Windows 7 64 bit on a 4GB notebook PC with tons of windows open e.g. a few Explorer windows open, a few Excel "windows"[1], a few Word windows, one Visio doc, Notepad++, Google Chrome, Firefox, putty, Outlook (a resource hog), Communicator, MSN Messenger windows, a Virtual Box Linux vm machine, Microsoft Security Essentials (it's my work PC so it's supposed to have AV) and it typically says 1700 to 2000MB _available_ (depending on how many firefox tabs, how many word docs and virtual machines etc). But overall no mem problem.

And guess which is using the most RAM? Not Virtual Box, not Word, outlook or Excel. It's Firefox with a 173MB working set and 142MB Private Working Set!

Yes it only has 500MB free memory, but so what? The O/S says there's 1700MB available. And so far I haven't had much slowdowns due to low memory issues.

To me the relevant metric for "low on memory" is "Pages Output/sec" (go launch perfmon.msc and add that counter). If that's a constant zero when you or the O/S switches from app to app, window to window, it means it's not swapping out. If it's not swapping out and not getting "out of memory" messages, it's not low in RAM no matter what some random "expert" thinks. And it's zero for me.

The equivalent in Linux for that is the swap "so" column when you run vmstat 1 (or vmstat 2). Same thing there - stuck at zero = not swapping.

I don't think my usage can be considered "light", as it is, what are those users running that's using up so much memory? Symantec or McAfee antivirus? ;).

FWIW, my laptop is not running any of the "OEM crapware" - I did a clean install of Windows 7 months ago when I got the laptop.

If that "expert CTO" can't even give an example of one memory hogging program (or show where Windows 7 itself is using so much memory that it's a problem), then it's likely he's full of crap.

Lastly, it's true my taskbar looks messy with two rows of task buttons, but I don't see the advantage of closing and reopening documents or programs if I'm not running out of RAM yet. I close them if I really do not need them (e.g. the document is out of date and not used for comparison). Otherwise it's much faster to just click a button to show the desired doc, rather than have to reopen it again from scratch (uses less battery power too - except in the case of MS Word which seems to use CPU even when "idle" - haven't figured that one out yet).

[1] By default Excel actually just has one window which changes to display the relevant document depending on which Excel taskbar button you click, whereas Word actually has separate windows for each doc.