86% of Windows 7 PCs Maxing Out Memory

CWmike writes "Citing data from Devil Mountain Software's community-based Exo.performance.network (XPnet), Craig Barth, the company's chief technology officer, said that new metrics reveal an unsettling trend. On average, 86% of Windows 7 machines in the XPnet pool are regularly consuming 90%-95% of their available RAM, resulting in slow-downs as the systems were forced to increasingly turn to disk-based virtual memory to handle tasks. The 86% mark for Windows 7 is more than twice the average number of Windows XP machines that run at the memory 'saturation' point, and this comes despite more RAM being available on most Windows 7 machines. 'This is alarming,' Barth said of Windows 7 machines' resource consumption. 'For the OS to be pushing the hardware limits this quickly is amazing. Windows 7 is not the lean, mean version of Vista that you may think it is.'"

Comments

When do people get this

[sopssa]

RAM is wasted when it isn't in use. The fact that the task manager in Windows says your RAM is used 95% tells nothing, and no it won't "result in slow-downs as the systems were forced to increasingly turn to disk-based virtual memory to handle tasks". I'm actually really surprised, and not in a good way, that "chief technology officer" of the company doesn't know this.

The new memory models in recent OS's try to utilize all the available RAM (as they should) to speed up things otherwise. It makes a lot of sense to cache things from hard-drive in low-peak usage points, and in such such way that it doesn't interfere with other perfomance. When the things that are most often used are already cached in RAM, their loading works a lot faster. This doesn't include only files, icons or such, but everything the OS could use or do that takes time.

If theres a sudden need for more RAM, the cached data can be "dropped" in no time. It doesn't matter if it averages at 25% or 95%, just that the perfomance overally is better when you utilize all the resources you can to speed up things in general.

Re:When do people get this

[Mr Thinly Sliced]

My understanding was that memory used for disk caching doesn't show up in task manager as "used".

It's been a while since I booted win7 though, so I might be mistaken.

Certainly under linux ram used as disk cache is marked "free".

It wouldn't surprise me that win7 has a heavier memory footprint though - as more applications move to .net and web browsers use lots of flash / silverlight etc - all of these things have a RAM cost.

Re:When do people get this

[eldavojohn]

The new memory models in recent OS's try to utilize all the available RAM (as they should) to speed up things otherwise. It makes a lot of sense to cache things from hard-drive in low-peak usage points, and in such such way that it doesn't interfere with other perfomance. When the things that are most often used are already cached in RAM, their loading works a lot faster. This doesn't include only files, icons or such, but everything the OS could use or do that takes time.

If theres a sudden need for more RAM, the cached data can be "dropped" in no time. It doesn't matter if it averages at 25% or 95%, just that the perfomance overally is better when you utilize all the resources you can to speed up things in general.

Assuming your claims of how Windows 7 is implemented are true, then the claims from the person who actually collected all the empirical evidence must be false:

resulting in slow-downs as the systems were forced to increasingly turn to disk-based virtual memory to handle tasks.

If the memory was freed up dynamically as needed then no processes would ever be forced to resort to disk-based virtual memory. So either you work at Microsoft and are assuring us that the implementation protects against this or you're speculating against someone who has claimed to gathered a large enough to make such accusations.

No offense but I'm going to side with the guy who appears to make his living testing these sorts of things ... the guy who is offering me numbers.

Re:When do people get this

[Anonymous Coward]

I think the issue here is that the system is turning to swap. Caching stuff that may be referenced again is fine and dandy, but if the system regularly turns to swap just to keep itself afloat, then you have a problem.

Re:When do people get this

[dr.newton]

From TFA:

"On average, 86% of Windows 7 machines in the XPnet pool are regularly consuming 90%-95% of their available RAM, resulting in slow-downs as the systems were forced to increasingly turn to disk-based virtual memory to handle tasks."

Re:When do people get this

[jibjibjib]

Using more RAM doesn't use more energy. Either your RAM is powered on, or it's not. And if it's powered on it maintains its contents, no matter whether the OS has actually written anything useful to it.

Re:When do people get this

[Sockatume]

If they'd measured pagefaults, they could've reported pagefaults. They didn't. RAM usage appears to be the total basis for the article, so his concern is a genuine one. We don't know enough about the study at this stage to dismiss it.

Re:When do people get this

[snemarch]

It shows up as part of the memory commit bar - which is what regular users will look at, and then go off screaming about "OMG IT USES ALL MY SYSTEM MEMORY!1!!! one one". It's also deducted from the "free" count, since technically it isn't free (it can be freed quickly, but has to be zeroed out before it can be handed off to a new app - security and all).

The Win7 task manager does show a "cached" stat, though, so your effectively free memory is "free"+"cached". And if you want more comprehensive memory stats, you should look at perfmon.msc or SysInternals' Process Explorer.

I wonder if TFA has actually measured that disk swapping happens (easy with procexp or perfmon), or are just shouting their heads off without understanding what's going on... it's well-known that SuperFetch utilizes otherwise unused memory for disk caching, and does so proactively :)

Re:When do people get this

[jimicus]

Here we see why /. needs a "-1, Wrong" mod.

Re:When do people get this

[phatcabbage]

Barth acknowledged that XPnet's data couldn't determine whether the memory usage was by the operating system itself, or an increased number of applications.
So yeah, it doesn't seem like the author really knows what's going on...

Re:When do people get this

[dhavleak]

If the memory was freed up dynamically as needed then no processes would ever be forced to resort to disk-based virtual memory.

The trouble is, the TFA doesn't actually say (at least not clearly) that the Win7 machines are indeed turning to swap more regularly. It just states that fetching stuff from the swap file is a consequence of running out of RAM and causes perf degradation. So if the Win7 machines are indeed utilizing all available RAM and yet not swapping at a significanly higher rate, it means they're making more optimum use of available RAM.

Re:When do people get this

[dunezone]

Hi there.

Ive been running Windows 7 since the BETA release. I have never experienced any issues that result in I/O thrashing against the hard drive as result of all my ram being utilized. I also have numerous friends running Windows 7 none have reported any issues like this, if anything its been praise for the operating system. So I am amazed to see a 86% number being thrown out there yet never seen this problem before.

No offense but I'm going to side with the guy who appears to make his living testing these sorts of things ... the guy who is offering me numbers.

And now lets quote something from the article...

Barth acknowledged that XPnet's data couldn't determine whether the memory usage was by the operating system itself, or an increased number of applications, but said that Devil Mountain would start working on finding which is the dominant factor in increased memory use.

This single sentence makes the article rubbish. They have no clue whats causing the heavy memory usage, its just an assumption that the OS is causing it and they're yelling fire before looking through all the data or completely analyzing the problem.

Re:When do people get this

[nmg196]

Totally agree. If you don't want Windows 7 to use the 4GB of RAM you've paid for to speed up your computer, take out 2GB and put it in the drawer. Otherwise, be thankful that it's actually making the most of the RAM you're using.

What next? People complaining that games use 100% CPU to give them maximum framerate when it could just use 30% CPU and give them 10 FPS?

Re:When do people get this

[snemarch]

Yep, that would be a problem - but neither the TFA nor xpnet mentions if this is actually happening, it seems that they're looking almost exclusively at "free physical memory", which isn't a useful stat in this regard. The xpnet site does say they factor in "how often it relies on virtual memory", but not how they do this (there's multiple metrics to choose from, some fairly uninteresting) and the fact that they seem to factor this in as a part of "memory usage" rather than keeping it as a separate stat makes me pretty wary of trusting any analysis from them.

Re:When do people get this

[lcarnevale]

HDD caching and swap are two completely different things. HDD caching is loading things from the disk to RAM to speed up things. SWAP is using the HDD as extra RAM when the system doesn't have any more memory left to use. So, what I think you wanted to do was to turn off swap, not hdd caching.

Re:When do people get this

[FooBarWidget]

Definitely the part about HDD caching slowing things down. Even in the DOS age it was well known that hdd caching utilities (I forgot the names, too long ago) improve disk performance tremendously.

Linux does the same things as Windows: it caches as much stuff from disk into main memory as possible. Try running:

    cat large_video_file.avi > /dev/null

You'll see that after running the command, your memory usage jumps up by the size of the video file. Now try running the same command again, it's now an order of a magnitude faster.

On Linux things like this are stored in main memory in the form of caches and buffers. I don't know about Windows, but Linux clears some caches and buffers if applications need real memory. Caches and buffers show up in memory usage reporting tools like 'free', so it's quite normal to see Linux systems using 90% or more RAM, most of which go to caches and buffers. It seems that most people who complain about memory usage don't know how memory is managed on modern operating systems, so they go all apeshit about "OMG HELP linux is using so much memory it sux0rz!!!" and I have to explain again and again how they're not getting it. Same goes to you. Now, Windows is suffering from the same problem.

FYI, here's the memory usage of my Linux server:

             total       used       free     shared    buffers     cached
Mem:           720        702         17          0         55        510
-/+ buffers/cache:        136        583
Swap:          399          0        399

It says 702 MB of used memory. Now look at "-/+ buffers/cache", it says 136 MB. That's the amount of memory *actually* used by applications.

Re:When do people get this

[cgenman]

Read TFA. It just claims that Windows 7 consumes all available RAM. That is the "empirical evidence." System slowdown was NOT measured.

Utilizing all available RAM is a pretty well understood technique at this point. All web browsers do this now, as do many other applications. One would expect a well-designed modern OS to do this. Consuming all memory itself is not a sign of poor programming itself, so long as disk caching of things that should be in RAM doesn't occur. This is not something that the people in the article has measured.

I'm going to side with the guy who appears to make his living testing these sorts of things.

Bad science is bad irrespective of the person conducting it. And whatever the original tester said is getting filtered through the viewpoint of the gentleman writing the article. Considering that he says that "Windows 7 is not the lean, mean version of Vista that you may think it is," yet never once compares statistics to Vista (or even mentions it outside of this statement), I'd take the conclusions from these stats with a grain of salt.

Re:When do people get this

[Moryath]

You obviously don't understand memory access design. It's all about feeding the CPU. There are two sorts of relationships we can use to make this work: temporal and sequential.

Hard drives are the largest-capacity storage (well unless you want to go to tape). But they're slow. Even the fastest high-RPM SCSI or SATA drives are SLOW compared to what's above them. This is mitigated, somewhat, by putting some cache memory on the drive's controller board itself. Still, having to "hit" the hard drive for information is, as you say, a slowdown. Same goes for "external" storage (Optical media, USB media, etc).

So you try to keep as much information as possible in RAM (next step up). Hitting RAM is less expensive than hitting the H/D in terms of a performance hit. In the original days of computing (up until the 486DX line for Intel CPUs), RAM and CPU operated on a 1:1 clock speed match, so that was that.

Once you factor in the "clock multiplier" of later CPU's, even the fastest RAM available today can't keep from "starving" the CPU. So we add in cache - L3, L2, and L1. the 486 implemented 8KB (yeah a whole 8K, wow!) in order to keep itself from starving. L3 is the "slowest", but largest, L2 is faster still but smaller, and L1's the smallest of all, but the fastest because it is literally on the same die as the CPU. That distinction is important, and in general you'll find that a "slower" CPU with more L1 Cache will benchmark better than a "faster" CPU with less.

The CPU looks for what it wants as follows:
- I want something. Is it in L1? Nope.
- Is it in L2? Nope.
- Is it in L3? Nope.
- Is it in RAM? Nope.
- Is it in the H/D Cache? (helps avoid spin-up and seek times) Nope.
- Crap, it's on the H/D. Big performance hit.

Everything except for the L1 check, technically, was a performance it. The reason for pre-caching things (based on temporal and sequential relationships) is all about predicting and getting what will be needed next into the fastest available place.

Yes, I suppose you can run an entire system where it all goes into "RAM", and you'll see it as "more responsive" simply because you never have to touch the hard drive. But turning off HDD caching is a BAD idea. It makes cache misses that much more expensive because then, instead of having even the chance of finding what you needed in RAM or in the HD's onboard cache, you have to wait for the H/D to spin up and seek to the right sector.

Re:When do people get this

[jimicus]

Maybe the part about HDD caching slowing things down?

I could be wrong there, since I'm not an expert but I remember the dark, dark days when my computer when spend 2-3 minutes just to redraw a Word document. Why? Because it was using the HDD like memory, instead of using the actual memory. It seems to me that this problem, while minimized, has never completely gone away.

Anyway telling me "you're wrong" doesn't enlighten either me, or the other readers. Please elucidate.

There's a lot of variables, but in simple terms the theory goes that something which you have recently accessed (be it an application, a document or whatever) you are likely to want again in the near future. Hence it's worth keeping a copy in memory on the offchance.

On the other hand, you really don't want to be swapping. So if a program needs more physical memory than what you have immediately available, it makes more sense to allocate memory which was recently holding cached data and just reduce the cache size than it does to start swapping, which is what any sane OS will do.

If there's any real intelligence involved in this, the OS will re-allocate an area which hasn't been used in a while.

The cache would only cause a problem in the way you describe it if the OS did not dynamically resize cache to account for other demands on system RAM.

I can't explain the differences between yours and your brother's computer but I can tell you that OEM builds of Windows tend to have so much garbage loaded at boot that they often need serious work before they're genuinely usable. Some of the builds I've seen, it is a wonder they boot at all.

Re:When do people get this

[snemarch]

actaully the windows 7 caching model is great. on games the difference between the first loading of a level and subsequent loads are night and day thanks to it's caching model.

That's the windows cache system generally, from way back in the NT days... Vista and later SuperFetch is more than that.

btw, regarding the article more directly: they shows no figure about the actual _swap_ usage, a thing that may or may not disprove their theory.

Indeed. The xpnet site does mention that they factor in paging somehow, but that's still pretty useless - paging activity needs to be a separate statistic. Also, simply looking at pagefile usage isn't terribly useful, an inactive app can have it's working set trimmed and pages flushed out to disk, and this won't matter much in the big picture.

What you need to look at is the rate of pagefile activity (ie., pages/second) as well as how often it happens - not just static numbers (even if having 1gig of data in the pf is probably a warning sign :))

Re:When do people get this

[Shakrai]

I think the issue here is that the system is turning to swap.

Turn swapping off. If you have a reasonable amount of RAM there's no reason to leave it turned on. I turn it off anyway for security considerations -- I use Truecrypt -- but really there's no reason not to do so on any XP machine with 2 gigs of RAM or Vista/Win 7 machine with 4 gigs of RAM.

Re:When do people get this

[afidel]

Actually what you really need to do is calculate hard pages/second which for some retarded reason isn't available as a default stat counter, ie those pages which actually go to the secondary backing store (disk).

Re:When do people get this

[afidel]

Windows gets really cranky when it doesn't have a pagefile. We tried it for performance reasons and we saw an almost 40% drop in performance despite the server not being under any kind of memory pressure.

Re:When do people get this

[TheLink]

Yeah I don't know why they don't set up the counter by default.

Anyway to set it up yourself:

Start perfmon.msc
Then add counters
go to Memory, add "Pages Output/sec".

I'm not an authority on virtual memory but from what I know:
Page Faults/sec is not usually relevant for this - the virtual memory stuff will have page fault even if it's not swapping to/from disk - it's part of how virtual memory works.
Page Inputs/sec could happen when you launch programs (then the O/S starts paging in the stuff it needs) - it's no indication of running out of memory.
Page Output/sec on the other hand is when the O/S is low and needs to copy stuff in RAM and write it OUT to disk so that it can reuse that RAM for something else. This is the one you want to monitor.

Re:When do people get this

[xigxag]

You missed the most important emphasis:

delays in I/O processing - ostensibly due to heavy virtual memory activity as Windows compensates for insufficient RAM. (emphasis mine)

Ostensibly means "to all outward appearances." In other words, they're admitting they don't really know the inner workings, the true cause of the delays. They're just supposing it's due to RAM swapping, as opposed to increased networking activity, aero glass, more concurrent programs being run on average, or any other number of other wag'd reasons. Basically, they picked two measurements that are both higher in W7, and just said, "Well, it stands to reason that A causes B." What's that phrase that internet smarty-pantses use all the time about this?

Re:When do people get this

[TheLink]

If stuff slows down due to that swap out, then it's still accurate enough for me.

Maybe the O/S could get it right and swap out and swap in Firefox in a way so I won't notice any slow downs. Give me an example of such an O/S please.

So far in my experience, if Windows or Linux swaps out Firefox for whatever reason, if I then switch to Firefox, I have to wait for it to be swapped back in.

Why "page out" and not "page in"?

"Page in" doesn't necessarily mean that I'll have to wait if I switch to different programs- the O/S is bringing stuff from disk to ram - I believe in some cases the O/S pages in stuff as part of running a new program - so it's not such a useful metric for "not enough memory".

But "page out" means something in RAM is going to disk - if I ever want it back in RAM, I'll have to wait.

If stuff in RAM is going to disk needlessly and causing unnecessary waits then the O/S virtual memory algorithm is getting things wrong.

Re:When do people get this

[Daengbo]

You mean your machine with 8GB RAM never hits swap? Wow. Shock. Color me surprised! O_o

Depends on what kind of memory

[A beautiful mind]

If it is filesystem cache, then it's not wasted or "maxed out". If it is application/system memory, then it is indeed a problem.

Re:Depends on what kind of memory

[A beautiful mind]

It's Windows. It might be a bug.

It's called SuperFetch

[dhavleak]

I guess Devil Mountain or whoever don't know about SuperFetch. Or need publicity.

And I guess slashdot editors don't know about SuperFetch. Or maybe an article like this gets them more traffic, revenue, etc.

The fucking bullshit that passes for articles these days..

Trollworthy rating for posting this...

[hitech69]

Computerworld should just close up shop for this worthless piece of journalism, or at least give their author the boot for doing any work with Craig Barth who represents a team of morons. samzenpus should be given a troll rating for getting this to Slashdot.

Page Faults

[Dr_Barnowl]

The metric to count is the number of page faults, an indicator of the number of times that the OS addresses memory that isn't in RAM.

As others point out, measuring just the fraction of memory consumption is stupid. I have 6GB of RAM ; my processes are using about 1.7GB of that, but the OS is claiming that 3.8GB is consumed. So that's 2.1GB of cached data that I no longer have to wait for from disk. Hooray.

TFA hints that they may be measuring page faults, and does mention that Win7 is hitting the disk for virtual memory more often. But they should make that clearer if it's the case.

Oh come on

[megla]

First we had submitters who didn't read the stories they were posting. Then we had editors who didn't read the stories they were approving. Now we have companies who don't read the articles they put out. Seriously, it's called a file cache. That's how it's supposed to work. Nice job, idiots.

Parent is +1 informative

[Anonymous Coward]

You cannot study virtual memory performance without considering how many page faults occur.

It is perfectly reasonable to use RAM as a filesystem cache, which is why Linux has adopted this approach. The effect is that almost all of the physical RAM is always in use. The cost is that pages are more likely to be wrongly swapped out - however, in typical cases, this increased cost is tiny in relation to the huge reduction in the number of disk accesses.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Available memory != Free memory

[TheLink]

Yeah. I don't have low mem problems with Windows 7. There's stuff I don't like about Windows 7 but "memory hog" is not on the list.

For work I'm using Windows 7 64 bit on a 4GB notebook PC with tons of windows open e.g. a few Explorer windows open, a few Excel "windows"[1], a few Word windows, one Visio doc, Notepad++, Google Chrome, Firefox, putty, Outlook (a resource hog), Communicator, MSN Messenger windows, a Virtual Box Linux vm machine, Microsoft Security Essentials (it's my work PC so it's supposed to have AV) and it typically says 1700 to 2000MB _available_ (depending on how many firefox tabs, how many word docs and virtual machines etc). But overall no mem problem.

And guess which is using the most RAM? Not Virtual Box, not Word, outlook or Excel. It's Firefox with a 173MB working set and 142MB Private Working Set!

Yes it only has 500MB free memory, but so what? The O/S says there's 1700MB available. And so far I haven't had much slowdowns due to low memory issues.

To me the relevant metric for "low on memory" is "Pages Output/sec" (go launch perfmon.msc and add that counter). If that's a constant zero when you or the O/S switches from app to app, window to window, it means it's not swapping out. If it's not swapping out and not getting "out of memory" messages, it's not low in RAM no matter what some random "expert" thinks. And it's zero for me.

The equivalent in Linux for that is the swap "so" column when you run vmstat 1 (or vmstat 2). Same thing there - stuck at zero = not swapping.

I don't think my usage can be considered "light", as it is, what are those users running that's using up so much memory? Symantec or McAfee antivirus? ;).

FWIW, my laptop is not running any of the "OEM crapware" - I did a clean install of Windows 7 months ago when I got the laptop.

If that "expert CTO" can't even give an example of one memory hogging program (or show where Windows 7 itself is using so much memory that it's a problem), then it's likely he's full of crap.

Lastly, it's true my taskbar looks messy with two rows of task buttons, but I don't see the advantage of closing and reopening documents or programs if I'm not running out of RAM yet. I close them if I really do not need them (e.g. the document is out of date and not used for comparison). Otherwise it's much faster to just click a button to show the desired doc, rather than have to reopen it again from scratch (uses less battery power too - except in the case of MS Word which seems to use CPU even when "idle" - haven't figured that one out yet).

[1] By default Excel actually just has one window which changes to display the relevant document depending on which Excel taskbar button you click, whereas Word actually has separate windows for each doc.

No kidding.

[endus]

Installed x64 on my 4gb machine and the performance was just ridiculously bad. I am a photographer and do a lot of image editing...couldn't even keep Cap One, Photoshop and iTunes open at the same time...especially if 7 was trying to thumbnail images in a folder (thumbnailing is broken and ridiculously resource intensive despite Microsoft's claims that its more robust in this OS). Noticed that it was swapping to the disk like crazy and ordered another 4gb. Definitely much better now, though I think 12gb would not be totally out of line for x64 with heavy applications. I haven't even TRIED to edit HD video yet...that is not a prospect I am looking forward to.

I pity people running the x86 version of the OS that are maxing out at 4gb. Definitely buy 64 bit (even though its more of a beta than a real OS) if you do anything memory intensive.

The one good thing about all this is that HOPEFULLY...FINALLY...maybe this will push Microsoft to push 64 bit more. They need to abandon 32 bit and force application writers and hardware manufacturers to start making 64 bit native applications. Working in a medical environment BELIEVE me I understand the need for backwards compatability, but the fact is that the resources are just not being put in to 64 bit to make it a really viable platform and even moderate power users are going to start bumping up against the 4gb limit.

Yea, I read that thing saying that the 4gb limit is a product-based limit rather than a technical limit but either way...it appears that x64 is where MS is choosing to support > 4gb so lets get serious about it.