Ars Analysis Calls Windows 7 Memory Usage Claims "Scaremongering"

Via newsycombinator comes a reaction at Ars Technica to the recently reported claims of excessive memory use on machines running Windows 7. From the article: "I installed the XPnet performance monitoring tool and waited for it to upload my data to see what it might be complaining about. The cause of the problem was immediately apparent. It's no secret that Windows 7, just like Windows Vista before it, includes aggressive disk caching. The SuperFetch technology causes Windows to preload certain data if the OS detects that it is used regularly, even if there is no specific need for it at any given moment. Though SuperFetch is a little less aggressive in Windows 7, it will still use a substantial amount of memory—but with an important proviso. The OS will only use memory for cache when there is no other demand for that memory."

Comments

So

[sopssa]

Though SuperFetch is a little less aggressive in Windows 7, it will still use a substantial amount of memory—but with an important proviso. The OS will only use memory for cache when there is no other demand for that memory.

I really wonder when people will get this. In the earlier thread I saw people commenting that Windows 95 didn't need so much memory and so on..

To state it again. This is not RAM memory you need, use or have purpose for. IF you do need it, it is zeroed-out and free'd to application in like 30ms (one frame in usual FPS games).

If you have fast memory, do use it to it's full extend.

Re:So

[Anonymous Coward]

Not everyone who reads slashdot has english as their first language. It's still understandable without inducing headaches in anyone who isn't a grammar nazi, it's fine.

Re:So

[Der PC]

You must be an anonymous moron making such a claim.

#1) Linux _doesn't_ have all the best and greatest technology built in.
#2) It even does have some really crappy technology in between.
#3) guess what the "-/+ buffers/cache" line in the output of "free" means
#4) guess what the "buffers" and "cached" columns of "free" means
#5) guess what prefetch/preload is.

Sheez... anyone gets access to a keyboard these days :-|

Re:So

[Amnenth]

People aren't taking the time to learn the meaning of the 'available' memory stat in the Task Manager.

Based on my experience, it's usually very close to the total you get when adding 'free' memory and 'cached' memory together.

'Available' in this case means that, as parent suggests, Windows will free it for use as soon as it's needed.

Re:So

[sopssa]

Linux (and actually all recent Windows, Mac OS X, Linux distros) use the very same technology. Linux did this really early too. I don't know the full story about intelligent RAM usage, but first time I saw it being used was on linux servers. And if you do compare it to the past "demand and receive, fully" memory model, it's a lot better.

Re:So

[Colin Smith]

It is basically.

You can even (trivially) roll your own in Linux. lsof occasionally to build some stats on commonly opened files. cat them to /dev/null to fill the filesystem buffer cache. I'm not sure anybody would even bother to give it a name like "SuperFetch", never mind a trademark like "Microsoft Windows SuperFetch®". It's kind of sad and depressing really.
 

Re:So

[ls671]

Yep, same for Linux. My Linux boxes use ALL the memory available even if I do not run many applications on it. The left over memory SHOULD be used as buffers/cache. If Windows 7 seems to use more memory from a newbie point of view, it might be because it does things like it should better than previous versions. I can't tell for sure since I have never tried win 7.

See this 4 GB Linux machine below, it only has ~49 MB of "absolutely free" memory and uses ~449 MB of swap.

In realty, it has ~2842 MB of "available memory" since it uses ~2792 MB of buffer/cache.

Using buffer/cache makes the system order of magnitude faster. If programs need that memory, the OS will give to them and use less buffer/cache.

free
                          total used free shared buffers cached
Mem: 4133252 4083380 49872 0 26852 2766248
-/+ buffers/cache: 1290280 2842972
Swap: 1999800 449244 1550556

Re:So

[Dutch Gun]

More to the point, the company that wrote this little monitoring tool badly misunderstood basic principles of how the operating system works. At this point, I think we can move on and completely disregard any conclusion they came to. It either demonstrated profound ignorance or a deliberate attempt to mislead people it what turned out to be a slashvertisement of their products and company.

From the article:

One might almost think that this whole exercise was simply a cynical ploy. Allegations of Microsoft bloatware are, of course, nothing new, and oblique references to the old canard that what Intel gives, Microsoft takes away does nothing to dispel the impression that this is another case of Microsoft bashing.

What a surprise. Fortunately, people really didn't even let them get away with it even in the previous article. Microsoft deserves plenty of what slashdot slings its way, but let's stick try sticking to facts.

Re:So

[ls671]

No disk activity is required to flush read buffer/caches. A little might be needed to flush write cache but read cache should be flushed first if memory is required by programs.

Anyway, most of the cache is read cache since write cache are flushed at regular interval.

Short story, no, no disk activity is required.

You will only have a lot of swapping if you do not have enough memory to run your programs, not because of buffer/cache usage. Before you start to have a lot of swap activity, your buffer/cache usage will go to zero anyway ;-))

Basically, if you do not have any buffer/cache usage on your system, it is probably because your are tight on memory.

Re:So

[rickb928]

"The OS will only use memory for cache when there is no other demand for that memory"

Ok, I'm not going to bother and read the smart people. I'm going to go straight to my point.

If you are using nearly all available RAM for disk cache, then EVERY REQUEST FOR RAM WILL REQUIRE CACHE DUMP.

It's like this;

If you have 4GB RAM and are using, say, 1.5GB for applications and system, and you use 2.2GB RAM for cache, then you are left with 300MB approx for any new demand. So any demand in excess is going to make your system dump cache, and it does take time. How much is an interesting discussion. And yes, we can consider if the request is going to be satisfied out of cache, but let's also assume if it is than that cache will not be dumped. That would be unfortunate.

Here's my beef with this much more aggressive caching in 7 v XP:

What the ^&)$ do you need 2.5G cache for?

What are the likely demands on caching? Office 2007? My XP machine at work rarely shows Office 07 components using more than 500MB RAM. 2G caching? Pagefile getting you down, you don't want to use that?

More to the point, how often would a real-world user be USING 2GB RAM, much less 4GB?

Well, I do. But real-world? I have 2-4 virtual machines running fairly often, and in XP I rarely get up to 3GB. Win7 would maybe get me into 4GB+, sure, and the pagefile would go crazy.

When Smartdrive first came out, we used to tune it down quite a bit to avoid hogging RAM. I know it's improved, and I've never worried about it since Win98, but the more I read about Win7's caching, the more I think it is there for some strategy here that has nothing to do with user demands, and everything to do with OS performance. Vista and 7 both cannot be descirbed as lightweight, so maybe this is really to keep the OS light on its feet.

Fine. But MS can't say so.

Another resource hog. Hopefully it won't hog anything we need, right?

I just don't get it. Obviously.

Re:So

[stumblingblock]

I have the time. So,pray tell, where do I learn the meanings of the various stats in Task Manager? Do you have a link?

Re:So

[BikeHelmet]

To state it again. This is not RAM memory you need, use or have purpose for. IF you do need it, it is zeroed-out and free'd to application in like 30ms (one frame in usual FPS games).

It's more like 100ms on an average PC, but yes, you are correct.

But since background stuff will be happening too, maybe 120ms...

If 120ms isn't an acceptable delay, then you need an OS where programs are geared for low disk IO usage, and low memory usage. That will prevent any software from interfering with any other software, giving very fast and consistent performance.

Selection of software is big. For example, the difference between My Uninstaller and Add/Remove in XP is huge. You wouldn't notice on a fast PC, but on an older one you would!

Superfetch is a crutch. A handy one, but it shouldn't actually be necessary to use it have great startup performance for your favourite apps.

Re:So

[Rockoon]

You seem to think that its not a read-only cache. SuperFetch caches disk blocks as they appear on the disk. The "dumping" of them means to not consider them as valid cache anymore. There is no need to write them out.

Re:So

[__aasqbs9791]

I'm not sure how you were modded insightful. Read the quote carefully. Eventually you'll understand it. Or die. Either way the world is better off*.

*This is a joke. Or not.

Re:So

[tagno25]

"those who fail to understand Unix are doomed to re-implement it...poorly."

lol.. like Linux "re-implemented" UNIX? Unless ofcource they used the same code, then the SCO lawsuit could be revived again ;)

They understand UNIX and re implemented it appropriately, not poorly.

Re:So

[newdsfornerds]

SuperFeltch FTW!

Re:So

[Blakey Rat]

The vast, vast, vast, vast majority of that cached memory is read-only caches (like DLL caching and superfetch) which doesn't need to be "dumped". Some small, very small, portion of it is read/write disk cache, but that portion is never going to be dumped unless you're *completely* out of memory otherwise. And that's basically a "last resort failure mode" at that point.

You're as bad as the guys who wrote that article in the first place. If you don't know how Windows works, please don't talk about it.

Re:So

[mystikkman]

You have no idea what you're talking about. Superfetch is much more than stats on commonly opened files. It takes into account the times of the day, weekends etc. too among other advanced stats. Anyway if it's trivial to roll your own, why doesn't such a thing run by default in Ubuntu? That's the thing that's sad and depressing, not giving names to technology.

Re:So

[Rockoon]

You're as bad as the guys who wrote that article in the first place. If you don't know how Windows works, please don't talk about it.

Hell, its not just windows. All operating systems do this.. and to be quite frank, programmers of all kinds should have cache techniques well understood. So the GP is neither a windows guru nor a decent programmer. The odds are very good that hes just an I-use-software geek, rather than someone who knows anything about computers.

Re:So

[Jah-Wren Ryel]

To state it again. This is not RAM memory you need, use or have purpose for. IF you do need it, it is zeroed-out and free'd to application in like 30ms (one frame in usual FPS games).

The problem with previous versions of windows (I haven't used anything newer than XP) is in how the OS decides that you do not "need, use or have a purpose for" certain types of memory.

The pathological, and yet all too common case with XP is the OS's decision that text pages should be dumped in favor of disk cache far too soon. The result being that if you have multiple apps open and a few that you haven't touched for roughly 10 minutes and then go to copy a couple of gigabytes of files around the text pages for those 'idle' applications are flushed out and the disk cache loaded with parts of those copied files (which you are unlikely to ever need). When you click on the iconbar to bring one of those formerly idle apps back to the foreground the system grinds away for a long time (obviously machine dependent but never instantly and frequently way beyond the point of annoying) as it reloads those text pages from disk before the application even starts to redraw itself much less starts becoming fully interactive again.

The worst part about that behavior is that, to the best of my knowledge, there are no knobs to tweak it. I can't specify how long a text page needs to be idle before it should be a candidate for flushing or even if it should be pinned down permanently so that is never paged out. I once went looking to see if there was a way to do it from within the application code itself - something like mlock()/mlockall() in posix - and I couldn't find an equivalent, which may just be a reflection of my own inexperience with the Windows API but I figured I would throw that out there anyway.

Re:So

[Again]

I have the time. So,pray tell, where do I learn the meanings of the various stats in Task Manager? Do you have a link?

rtfm... Oh wait... This is Windows. I could never find the man pages for that system.

Re:So

[cbhacking]

One of the main tricks, which you're completely glossing over, is the algorithm that determines what files are likely to be wanted/needed at any particular time of day. Sure, lsof will give you the raw data on what's open when, but making something useful out of that it a touch trickier than you suggest.

Additionally - and forgive my lack of Linux kernel expertise; I never thought to check this - is there a limit on the size of the filesystem cache? If it's less than a couple gigabytes, you're not going to be able to pre-load some of the files that most need it. If it is that large, presumably the ability to re-allocate it to a user-space program near instantly is also present, but that would be my next question. Finally, is the cache FiFo? If so, you've got a problem - unless your users are literally as regular as clockwork, they aren't going to want program X, and nothing but program X, at exactly time Y every day. Therefore, to ensure it's pre-loaded for them, you have to cache it in advance. However, if there's a lot of filesystem access - even small everyday stuff, like browsing the web or looking through a photo collection - the constant disk I/O will shove a lot of your pre-cached data out of the buffer.

Superfetch doesn't *sound* terribly impressive, but more thought went into it than you might realize.I'm certainly not saying it can't be duplicated on Linux, but there's obviously a reason it hasn't been, and it's obviously not because the feature is useless. I'm not sure how relevant it is to servers, but from a user perspective, the difference between five seconds and under one second to open a particularly large program is extremely noticeable. Given enough RAM (4GB on my system, of which Superfetch typically is using upwards of 3) you can easily achieve speedups like that.

Re:So

[Anonymous Coward]

Citation needed.

Re:So

[dhavleak]

To state it again. This is not RAM memory you need, use or have purpose for. IF you do need it, it is zeroed-out and free'd to application in like 30ms (one frame in usual FPS games).

It's more like 100ms on an average PC, but yes, you are correct.

Err.. what is more like 100ms? Where are you getting these numbers from?

Superfetch is a crutch. A handy one, but it shouldn't actually be necessary to use it have great startup performance for your favourite apps.

By design, your favorite apps would be precisely the ones to benefit from SuperFetch.

Re:So

[berboot]

SuperFetch also keeps track of what times of day that applications are used, which allows it to intelligently pre-load information that is expected to be used in the near future.
Source: Wikipedia
Their work, says Horvitz, was able to predict which applications users would open by time of day and also by day of the week.
Source: InfoWeek

Re:So

[vadim_t]

Recent versions of Ubuntu preload some stuff on boot at least. For doing it during runtime there's the "preload" package. Now the good thing is that if I think it's pointless, then I don't need to install it, and it isn't on my system.

And I concur with the grandparent, the naming of various mostly not very interesting technologies with some moniker to make it sound more exciting is annoying, and I enjoy the lack of that stuff in Linux.

Re:So

[ianezz]

I'm certainly not saying it can't be duplicated on Linux, but there's obviously a reason it hasn't been, and it's obviously not because the feature is useless

Have a look at preload (project registered in 2005). I believe it's installed by default on Ubuntu. YMMV. HAND.

Re:So

[SScorpio]

By design, your favorite apps would be precisely the ones to benefit from SuperFetch.

Exactly, I still can't believe all the complaints people have against Superfetch ever since Vista came out. The whole purpose of it is that it monitors your computer usage and keeps copies of commonly used DLLs and applications in memory. So when you want to start one of your favorite apps, the files it needs are already in memory and it can start instantly. Without Superfetch you'd be reading the files off the hard drive which is much slower.

If Superfetch is so terrible I wish the people complaining would just disable the service and then complain how slow it starting applications are on their computers.

Re:So

[mystikkman]

Superfetch is a crutch. A handy one, but it shouldn't actually be necessary to use it have great startup performance for your favourite apps.

Huh? Superfetch adds to whatever great startup performance apps have, I don't see how it's a bad thing. It's more like a turbo than a crutch, adding to existing speed.

Re:So

[Sir_Sri]

There's probably a psychological threshold. If most linux desktop apps are small, say a couple of hundred mebibytes, then it's going to take say 15 seconds to get the data from disk and 5 or so to load it up into memory, if you cache it in ram, and it now takes 1.5 seconds to access in memory and 5 to process (you still have to move the data to the processor) you've got from a system that takes say 21.5 seconds to load to one that takes 6.5 to load or roughly 1/3rd. Yes that's all canned off the top of my head. It's noticable, but people are tolerant of 15 seconds here or there, but it's a lot of development time to trim that down.

A really big app, (say world of warcraft), which takes 500 or 600 megs in memory - but a different 500 or 600 depending on where you are, cannot be all that effectively cached by the OS, so it's the same slow loading either way.

In that sense there's probably fairly small set of apps that users would notice and care about a feature like prefetching. It's not bad to have, but it's not critical. Probably you go and you set some thresholds, we want to load small applications that currently take one minute to load to load in 10 seconds, but applications that take 2 minutes to load, well we can't do anything with anyway, and applications that take 30 seconds to load will now load in 8 maybe, and that sort of thing. The more memory you have the easier that is, but the more memory you have the more memory big apps (like the future equivalent of world of warcraft) will use, and you're not, in terms of average load times, any further ahead.

On a server, the applications you want are usually already running, (if they aren't running odds are the server is rebooting so caching hasn't helped you a whole lot), and they need to handle caching their own data in their own way, the OS doesn't know which webpages are most likely to be served by your web server application sort of thing.

Re:So

[Guspaz]

http://www.linuxatemyram.com/

Re:So

[m_pll]

So,pray tell, where do I learn the meanings of the various stats in Task Manager?

You can press F1 while in task manager and then search for a particular metric, e.g. "available memory". This produces results that seem moderately useful, for example:

Under Physical Memory (MB), Total is the amount of RAM installed on your computer, listed in megabytes (MB). Cached refers to the amount of physical memory used recently for system resources. Available is the amount of memory that's immediately available for use by processes, drivers, or the operating system. Free is the amount of memory that is currently unused or doesn't contain useful information (unlike cached files, which do contain useful information).

For more details about particular counters you can check the Windows Internals book, or Memory Performance Information on MSDN. Also, many counters in task manager have similar or identical perfmon counters, and perfmon has its own help (IIRC there's a "show description" option in the counter selection dialog)

.

Re:So

[wish bot]

It was only a couple of years ago that one of the biggest criticisms of OS X by Windowsers was "it's a memory hog". Explanations about pre-caching fell on deaf ears.

Welcome to 2003 guys!

Re:So

[X0563511]

Sadly, superfetch does more than just preload. it tracks usage and predicts what data will be requested at a given time. Yes... it knows you often load up Outlook at 9am on Monday, but at 10am on Friday.

Re:So

[cprice]

Linux does something very similar to SuperFetch from the sounds of it - I have no experience with, nor do I want to run Windows 7. I believe Linux has been doing this since 2.2 or 2.4 kernel came out.

Re:So

[vadim_t]

preload monitors applications that users run, and by analyzing this
  data, predicts what applications users might run, and fetches those
  binaries and their dependencies into memory for faster startup times.

Still, I never found an use for that kind of thing, so I don't bother installing it.

Re:So

[quadrox]

Problem is, developers get used to not having to worry about memory consumption and performance.

Re:So

[toadlife]

Cutler personally hated UNIX. NT was his anti-UNIX.

Good reading here.

Re:So

[Ranzear]

Its an excuse for Linux users to brag about how much RAM they're not using. In the age of 'zomgz, Vista uses >1Gb or ram!!' I loaded up some video editing, piled in as much media as possible until the system was strained to breaking, and then killed the process: Vista was using 297Mb.

Re:So

[Nutria]

Lots of people hated Unix. They even wrote a book.

Re:So

[j00r0m4nc3r]

I prefer the term turbocrutch

Re:So

[rickb928]

When I say dumped I mean dumped, discarded, tossed.

If I thought the read cache needed to be written out, I would be aghast. I do know how caching works. I just think the algoeithm win7 is using os too aggresssive.

I notice it when our testbeds get low on RAM. They take longer to come around than when we run XP.

Re:So

[Antiocheian]

superuser and less-privileged users - implemented since win2k

What taste does your thumb have ?

Re:So

[ozmanjusri]

So does Microsoft.

Re:So

[ymgve]

Just turn off the pagefile.

Windows can't page when it doesn't have a pagefile. Problem solved. At least it works for me.

Re:So

[AHuxley]

Quality SSD would help with this I hope.
As for RAM use, most modern OS seem to love more ram and use it, keeping it full.
Does MS give the RAM back to apps in strange ways?
Does MS stutter or have ram use bugs?
My thought about ram is get as much as you can afford/can be used for a 32/64 bit OS and your max tasks.

Re:So

[AHuxley]

On OS X if you have 1 or 2 gb of ram, lots of page in and outs with the HD hits and slow feeling. Add 4 gb and they should stop under the same usage.
MS seems to keep touching the HD no matter the ram installed??

Re:So

[AmberBlackCat]

rtfm... Oh wait... This is Windows. I could never find the man pages for that system.

Press F1...

Re:So

[ls671]

> never needed swap space.

You, sir. misunderstand the basic concepts of an OS and thus the relevance of setting up a swap space.

Swap is used by my machine to swap out process that I haven't used for days as well as part of processes that haven't accessed the swapped part of their memory for days. I almost have no swap activity (exchange between disk swap and memory).

I am grateful that Linux is smarter than you seem to be, it figured out that it was more efficient to swap out those process and to regain the memory to use it as buffer/cache.

My system would run fine without a swap space, it would just run less efficiently, with less buffer/cache.

http://slashdot.org/comments.pl?sid=1557492&cid=31213314&art_pos=5

> I see you have like 2GB of swap, haha (moron). Your machine would grind to a halt thrashing the disks if you swapped out that much memory.

2 GB is the correct size for my setup to allow a safety margin for memory leaks and what not and to maintain an optimal amount of buffer/cache at all times. Linux would automatically swap zombie assigned memory that isn't used anymore. As a matter of fact, it probably already has ;-))

Re:So

[Foolhardy]

I once went looking to see if there was a way to do it from within the application code itself - something like mlock()/mlockall() in posix - and I couldn't find an equivalent, which may just be a reflection of my own inexperience with the Windows API but I figured I would throw that out there anyway.

The function you're looking for is VirtualLock. You may also look into increasing the process's minimum working set with SetProcessWorkingSetSize. This requires SeIncreaseBasePriorityPrivilege.

A process that is scanning through a file is supposed to use the FILE_FLAG_SEQUENTIAL_SCAN hint so that the cached pages are recycled first, but that doesn't always happen. It also doesn't help that csrss will ask the kernel to minimize a process's working set when its main window is minimized.

Re:So

[nabsltd]

This is not RAM memory you need, use or have purpose for. IF you do need it, it is zeroed-out and free'd to application in like 30ms (one frame in usual FPS games).

The reason that SuperFetch in Vista degraded performance is that it was too agressive about keeping memory filled.

For example, when the game you are playing asks for more memory (to load a video between levels, for example), Vista did give up the cache fairly quickly, but as soon as the game released the memory, Vista started reading more stuff from the hard drive to fill that memory back up. 99% of the time, what it was reading wasn't the right thing. So, when then game went to load the next level and again asked for some temporary extra memory, Vista would have to stop filling the cache, zero out the memory, and seek the hard disk to the right place.

Basically, there's nothing wrong with a smart, unobtrusive prefetch algorithm that fills up RAM as cache. SuperFetch on Vista was neither smart nor unobtrusive. Hopefully, the Windows 7 implementation is a lot better.

Re:So

[Anonymous Coward]

You never found a use for wasting less time? Seriously?

Re:So

[m_pll]

Starting with Vista, working sets of GUI processes are no longer emptied when the main window is minimized.

For the standby cache recycle problem, Superfetch can help a lot. First of all, it can detect when apps do things like read lots of files sequentially without using FILE_FLAG_SEQUENTIAL_SCAN (or when they do this through a mapped view) and deprioritize these pages so they don't affect normal standby memory. And if useful pages still end up being recycled (e.g. because some app temporarily consumed lots of memory), Superfetch can re-populate them from disk later.

Re:So

[m_pll]

Turning off the pagefile will not prevent DLL/EXE pages from being paged out, which is what the the person you're replying to complained about.

Re:So

[kangsterizer]

the OS is doing the caching (including disk read/write) and its doing so more efficiently that you can with a user space process.,For most programs, you dont need to do this kind of caching yourself.
For some, it might benefit and generally used.

most of the missing things in optimization are proper code, proper code path, and proper code design. (yeah ok, "proper code" sums it!)

Re:So

[Randle_Revar]

>My Linux boxes use ALL the memory available even if I do not run many applications on it.

Really? Is that a Gnome and/or KDE thing (seems like that would be odd)? I only have 188MB out of 4GB free, but that has a lot to do with having way too many tabs open in 2 Firefox instances and 1 Mozilla instance. When I start up and it's just Awesome WM and my basic programs, memory usage is very low

Re:So

[toadlife]

I own a print copy with the original barf bag still attached. :)

Re:So

[ls671]

> Really?

Yes really.

It depends on the application you are running although. It is mostly related to how much disk content you access. If you access more disk content than your total memory, all your spare memory is going to end up as buffer/cache.

All you have to do is run a webserver (no GUI programs) with ~4 GB of content that get accessed to end up using all your memory on a 4 GB machine.

Rsync processes to back up all your disk content are pretty good at it too. Heck, maybe copying a few very large files would do the same.

> but that has a lot to do with having way too many tabs open
> in 2 Firefox instances and 1 Mozilla instance.

This has nothing to do with buffer/cache usage. Have you tried running "free" on your machine ? It should speak by itself...

Re:So

[Randle_Revar]

Oh, I see. I never really paid attention to the "cached" section, and I was thinking more of desktop usage, rather than server. Although I do have about a gig of cached stuff right now.

Re:So

[BikeHelmet]

Err.. what is more like 100ms? Where are you getting these numbers from?

The time to zero out 2GB of memory?

It's a pretty simple calculation. Divide the amount of RAM(GB) by your RAM bandwidth(GB/sec) and multiply by 1000(1 sec) to get the miliseconds it might take. Oh sure, there's probably CPU optimizations to speed it up - but your OS is still running and doing other stuff in the background, and has to mark down the pages as free. Those are extra tasks that also take time.

The net result is a guesstimate, but will be pretty close to how long it takes, and is significantly above 30ms unless you have a ton of memory bandwidth.

For a worst case scenario, where there are no optimizations, an average dual-channel DDR3 1600mhz PC would take ~170ms (excluding OS overhead) to free 2GB of RAM. I'm guessing the optimizations are quite extensive, but not enough to hit 30ms to free 2GB. On an older DDR2 machine, it would take even longer.

Perhaps someone else that knows more about CPUs and memory could chime in? A Sun/Java engineer? :P

Re:So

[Nutria]

I am sooooooo jealous!

Re:So

[nhytefall]

RTFM, or use Google. Don't be a sarcastic moron when you obviously fail at having the skills to do so.

Re:So

[toadlife]

I bought it from Amazon, used, about three years ago and only paid about $15 for it. It's not mint condition, but decent - looks like it has been read. You could probably still find it on ebay or amazon.

Don't let my sig fool you. I'm actually quite fond of UNIX. I wish I could use it at work more. :(

Re:So

[siride]

Swap space is only for anonymous pages, essentially, those that are private mappings, or shared, but not mapped to any part of a file in the filesystem (e.g., BSS sections for libraries). The rest: code and read-only data can safely be discarded and reloaded from the original file. So unless you are using a huge amount of private anonymous pages, you won't see any swap partition activity. I know when I ran Linux I almost never hit the swap partition. That is, it never even bothered to put stuff there in the first place, much less access it again. Rather than waste disk space on a rarely used resource, I got rid of it. The cost of having to keep a few more page frames active is negligible.

Re:So

[siride]

Do you think dumping a page frame costs that much? At worst, you have to zero it out, but if you know what's going in it when it's time to dump, then you can simply copy the page from the HD into the frame and muck about with a few tables in the kernel and that's it. It shouldn't be non-trivially more expensive than filling an already free page frame. The worst possible extra cost would be modifications to additional tables related to the cache.

Re:So

[ls671]

> Although I do have about a gig of cached stuff right now.

I would say this is about the minimum to maintain good performance on your system (4 GB system you said) if all your memory is in use. When you run low on buffer/cache usage and that all your memory is used, it means that your are tight on memory.

I had a customer which did not believe he needed to put more memory on his server because it was barely swapping at all. His server could barely use any buffer/cache memory although. He doubled the memory on his server and the machine was able to handle 5 to 10 times more load ;-))

80% or more of your memory used as buffer/cache is great on web servers in order to boost performance.

Re:So

[man_of_mr_e]

In 2003, 4GB of memory was pretty expensive. Today, it's less than $100.

Re:So

[siride]

And the fact that zero-ing pages is not done immediately, but is done as a low-priority process afterwards. Still, if you have a problem with freeing 2 GB of RAM in under 120ms, then you might want to re-evaluate your usage patterns, or get a much faster machine.

Re:So

[ls671]

Can you please confirm that my system has swapped a large amount of anonymous pages e.g. ~449 MB. I am very curious about this.

Please also note that my system uses ~2792 MB of buffer/cache as shown in the free output linked below. My conclusion is that since the system uses ~2792 MB of buffer/cache, it didn't swap ~449 MB of data because it was tight on memory. It would have regained memory for the buffer/cache memory before that.

My understanding is also that if I didn't have a swap partition the ~449 MB swapped out would need to be subtracted from the~2792 MB of buffer/cache the system is using, thus having the system work with less buffer/cache.

So in my case, it seems my system DOES put stuff in the swap space and this is what I have been used to see for quite a while.

http://slashdot.org/comments.pl?sid=1557492&cid=31212938&art_pos=11&art_pos=11

Thanks,

Re:So

[Artefacto]

That's strange, I find that behavior much worse in Linux (i.e., Linux is more aggressive swapping out applications). I usually tune the "swappiness" (see http://kerneltrap.org/node/3000 ) to very low levels, but sometimes after having to wait a few seconds each time I switched to some huge application such as NetBeans made me disabled the swap space.

Re:So

[dhavleak]

Err.. what is more like 100ms? Where are you getting these numbers from?

The time to zero out 2GB of memory?

I still don't get it -- what operation/action/circumstance does this represent? Why calculate the time to zero out memory, and why 2GB of it? It has nothing (absolutely nothing) to do with the topic at hand (superfetch, page-faults, page-ins etc.).

Re:So

[KillShill]

Superfetch can be turned off... it's implemented as a service and is easily disabled.

Re:So

[Anonymous Coward]

If they don't have to worry about it, then it's not a problem.

Re:So

[BikeHelmet]

People like to say Superfetch has no effect on the start speeds of unfetched programs.

If starting a game - many of which could easily consume 2GB - Superfetch could possibly delay the game starting by a quarter to a half second.

If you have a regular HDD, and the game takes 15 seconds to load, this would not be noticeable. If you have an SSD, and the game takes 3 seconds to load, this might be noticeable. I believe this is why OCZ recommends disabling Superfetch for their SSDs.

I'd say it's relevant, because understanding what's going on lets us make more educated decisions as technology changes.

Re:So

[Nutria]

You could probably still find it on ebay or amazon.

Good thought.

I'm actually quite fond of UNIX.

Not me. I've been able to stick with OpenVMS for most of the last 20 years!

I wish I could use it at work more.

The future where I work is Oracle on Linux. That fleet of supertankers has sailed and there's no altering course. :(

I've been using Linux as my sole desktop for 10 years (Debian now for 7 years), because it gives me the CLI power I expect from an OS. What I'd prefer is OpenVMS which kept up with the times.

Re:So

[rastoboy29]

Turn off swap.  I never run Windows with swap enabled these days for just this reason.

In these days of a minimum of 2 gigs of RAM, for almost all applications it just doesn't matter.  I even do 3d modelling in Blender without swap.

Obviously not an option if you're cutting it close in RAM usage, though.

Re:So

[dhavleak]

Your conclusions are incorrect - you seem to not get a few things about how caching works.

Primary among them - when you page-fault, and then page-in, you do not need to zero-out that page. This is why I said your calculation for the time required to zero out 2GB of memory has no relevance to the topic at hand.

I'd say it's relevant, because understanding what's going on lets us make more educated decisions as technology changes.

I'm not trying to be snarky -- I didn't get the relevance because I couldn't understand why on earth you're calculating the time to zero out 2GB of memory. Understanding what's going on is certainly important -- I suggest some reading up on page faults and superfetch. I don't know about this OCZ article (link would be nice) but I am very very skeptical. Turning off ReadyBoost when you have an SSD makes sense. Turning off SuperFetch when you have an SSD, makes zero sense. SSDs are fast. They're still very very slow compared to RAM.

Re:So

[vadim_t]

I don't spend a significant amount of my time launching applications. I start things maybe once a week, and just keep suspending and resuming my laptop. Most of my time is spent in 3-4 apps that are constantly running already.

Also my usage isn't all that predictable, so it probably wouldn't work so well.

Re:So

[gtall]

A lot of DEC people and users hated Unix. I could never figure out why exactly other than they had a lot of time and effort invested in VMS and Unix was growing at the time. They felt threatened because Unix did a lot of things differently. I recall VMS being a quite capable OS. It was never intended for small computers, it doesn't surprise me that Culter produced a NT which wasn't suited to small computers given that MS had to drill holes in it to get acceptable UI behavior. Java had/has the same problem, it was developed by a Big Machine people and as a consequence, their UI stuff still sucks.

Re:So

[mikechant]

Have a look at preload [sourceforge.net] (project registered in 2005). I believe it's installed by default on Ubuntu.

Not installed by default (on 9.04 anyhow) but I've found it pretty useful with its default settings. Frequently used applications seem to load in about half the time the first time they're used after a reboot.

Re:So

[Foredecker]

mmmm... they are here. http://msdn.microsoft.com/en-us/default.aspx

Re:So

[siride]

Yes, your system has swapped large numbers of anonymous pages. My system never did, but my usage patterns are probably different from yours. However, it's not so easy to say "well, the stuff was swapped out to make room for the cache, which is always a good thing". It isn't necessarily. You only have so much RAM, so when stuff is swapped out, or cache is trimmed, something is going out to disk. And either way, you are going to have to go back to disk to get it. The question is whether you are more likely to need to go and get no longer used files, or not recently used anonymous pages. And that, of course, depends on your usage.

Re:So

[Rockoon]

You are making the assumption that caching only applies to disk drives. You will remain a second rate programmer until you learn that algorithms dont apply to single problems. Sometimes you will cache data from a drive, sometimes you will cache data from a network, and sometimes you will cache pure computation.

Re:So

[drsmithy]

For a worst case scenario, where there are no optimizations, an average dual-channel DDR3 1600mhz PC would take ~170ms (excluding OS overhead) to free 2GB of RAM.

Where is the data going to come from to fill that 2GB, such that ~170ms is a meaningful delay ?

Re:So

[BikeHelmet]

Primary among them - when you page-fault, and then page-in, you do not need to zero-out that page. This is why I said your calculation for the time required to zero out 2GB of memory has no relevance to the topic at hand.

Oh? I was under the impression it was a security requirement for Windows. Something about not letting new processes sniff what was previously held in those memory pages.

http://www.ocztechnologyforum.com/forum/showthread.php?47212-Vista-32-64-SSD-Windows-Registry-tweaks

SSDs are quick where it counts when loading a game - access times, and read speeds. Most games load a few hundred megabytes off the HDD, and decompress those textures and levels into thousands.

Lets say for argument that a game only needs to load 200MB, then decompress it. The SSD would take about 1 second to load that data, and the CPU would take a while to decompress it.

If you're burdening your CPU with zeroing out pages, it could cause the loading to take longer. As demonstrated, perhaps as much as a half-second longer. When your load times are so short already, this is actually a noticeable amount of time.

If zeroing out pages does not in fact apply, then I'm sorry for making an invalid argument.

Re:So

[drsmithy]

So, when then game went to load the next level and again asked for some temporary extra memory, Vista would have to stop filling the cache, zero out the memory, and seek the hard disk to the right place.

This would account for maybe 50ms of time. Are you seriously trying to suggest half a second would meaningfully impact how quickly the next level in a modern game loads, when it probably has to pull dozens or hundreds of MB of data off a disk at ~20-30MB/sec ?

Re:So

[BikeHelmet]

Probably decompressing textures or level data. Even highly optimized, though, that would still take multiple seconds.

But I wasn't arguing against it. I said Superfetch was handy. I was just also pointing out that the numbers were off. If it were RAM usage or CPU usage, and the amounts were off by 600%, you'd understand. :P

So sorry for being anal! I seem to have started an interesting debate, though.

Re:So

[Blakey Rat]

When I say dumped I mean dumped, discarded, tossed.

I thought you meant it had to be saved to disk first. Sorry.

But that brings up another question: how long do you think it takes to dump RAM? (Without saving it back to disk?) Honestly, it's something the CPU can do with spare cycles between actual useful work anyway... for all practical purposes, dumping RAM takes *zero* time.

And that was the whole point I was trying to make. If you can dump RAM to use for another purpose in practically no time, then it's retarded to keep RAM free instead of using it for aggressive pre-caching.

I notice it when our testbeds get low on RAM. They take longer to come around than when we run XP.

Well, if you're using the computer as a testbed, then pretty much by definition you're not doing what the normal average user is doing. Windows is optimized for normal human beings, not for stress testing. I don't see your experience here as being particular valid when debating the usefulness of caching.

Re:So

[drsmithy]

It was only a couple of years ago that one of the biggest criticisms of OS X by Windowsers was "it's a memory hog". Explanations about pre-caching fell on deaf ears.

OSX was not doing this in 2003 as far as I know (and if it was, it was doing it terribly given its dismal overall performance). Do you have some supporting documentation ?

Re:So

[drsmithy]

Superfetch doesn't *sound* terribly impressive, but more thought went into it than you might realize.I'm certainly not saying it can't be duplicated on Linux, but there's obviously a reason it hasn't been, and it's obviously not because the feature is useless. I'm not sure how relevant it is to servers, but from a user perspective, the difference between five seconds and under one second to open a particularly large program is extremely noticeable. Given enough RAM (4GB on my system, of which Superfetch typically is using upwards of 3) you can easily achieve speedups like that.

You've pretty much answered your own question. SuperFetch primarily benefits interactive desktop users, a market Linux does not play a significant part in.

Re:So

[rickb928]

Our testbeds do UAT for our newest app. Manual input, no scripts. We reimage them for different tests, and sometimes save an image for later use.

And these don't live in our AD domain. So they are simplier and get less antivirus.

Other than running our .NET/SQL app, these are simple machines.

My desktop is what gets a lot of VPC use.
 

Re:So

[kangsterizer]

except i CLEARLY gave disk as example since its the most common thing to be cached

caching a computation usually means simply keeping the result in a variable that you don't destroy
if you're however talking php and stuff that's a pretty entirely different matter and actually solved efficiently by caching the entire page most of the time (did you read *most of the time* or you're too retarded?)

i think you will remain a second rate human until you read properly

Re:So

[dhavleak]

Zeroing pages is done for free memory for allocations, for security reasons -- yes. It's not the same thing as paging-in, where you're about to overwrite the entire contents of the page -- so there's no need to zero it out first.

Re:So

[quadrox]

Oh yeah, because as long as we have enough landfills to dump all our waste, it's not a problem that we're wasting many resources on crap. Just because you CAN do something, doesn't always mean you should.

The faster my computer is, the better, but if programs just get slower every time I upgrade my hardware, then wtf are we improving hardware technology for? So we can have more lazy developers?

Re:So

[Rockoon]

Basically what you are saying is that you have no use for caching algorithms even though you don't know anything about caching algorithms.

Thanks for proving my point.

Re:So

[thePowerOfGrayskull]

Though SuperFetch is a little less aggressive in Windows 7, it will still use a substantial amount of memory—but with an important proviso. The OS will only use memory for cache when there is no other demand for that memory.

I really wonder when people will get this.

If it were correct, I would "get it". But when tabbing back to my app which has been sitting idle for 30 minutes causes page file reads, and at the same time I can see that there's a signficant amount of "cache" in use, this means that the OS is using the cache at the expense of applications that need the memory. I don't understand how people can consistently claim that this is only done when there is "no other demand" for the memory, unless they think that because an application is idle, the OS should be reclaiming its memory.

Another way of looking at it: Win7 is using SuperFetch to cache the applications I *might* want to use in priority over the memory of the applications I *am currently running* -- even if not actively so.

When I have 4GB of RAM in my box, and I am seeing pagefile activity when running apps consuming a total of 2GB, there's something very wrong.

Re:So

[cbhacking]

Neat, and thanks. I'll look into getting it on my Linux box. This is one of those little features that make so much difference in user experience; the less time you have to look at a busy mouse cursor while loading things, the easier it is to convince people to use the software.

Re:So

[cbhacking]

While this may be true today, desktop Linux is gaining traction, and especially is gaining mindshare. The more people who know of it, and are willing to try it out, the more features like this are going to matter. When somebody tries Linux, if they notice that it's really snappy loading a program, that is going to positively influence their opinion of the OS and hopefully they'll spread that positive opinion. This is exactly the kind of positive feedback loop Linux needs for gaining desktop acceptance, even if it requires writing code that is of limited use in the current install space.

Conversely, if it seems to take a lot longer to do something - especially something Windows does quickly - potential Linux converts are less likely to recommend Linux to others. Random example: Firefox takes bloody ages to load on my school's Linux lab machines (5 seconds would be fast; I've seen 15). It's probably a misconfiguration on the part of the support staff; the computers are plenty fast enough. However, the first impression of *everybody* on using them is "Why is loading the web so damn slow?!?" and many aren't going to have previously seen Linux anywhere else. That's not the kind of first impression we want to give.

Re:So

[makomk]

No, Linux doesn't. It caches data from disk if an application requests it, and it uses all available RAM to do so. However, it specifically doesn't fetch any data not requested by an application (with the exception of reading more data than the application has requested if it expects that data to be used immediately).

Re:So

[nabsltd]

This would account for maybe 50ms of time.

With a well-designed multi-tasking system and cache, you're probably right (although 50ms is 1/20 of a second, not half a second), although with hard disk seek times of 10-15ms (for real-world average hard drives), 50ms might be a low estimate.

But the experience that people had with Vista SuperFetch is that it thrashed the disk constantly (which it shouldn't...once RAM is loaded, the disk should idle), and seemed to do stupid things like write out pages to the page file in favor of loading that RAM with "predictive cache". You can see how this would really slow things down.

But, even without this, a "predictive" cache should still put more weight to recently used pages from the active application, especially if there hasn't been any change of foreground app in a while (which is likely with a game). This would result in a user experience which is better by being more responsive, even if it's only skipping a few delays of 100ms.

Re:So

[drsmithy]

With a well-designed multi-tasking system and cache, you're probably right (although 50ms is 1/20 of a second, not half a second), although with hard disk seek times of 10-15ms (for real-world average hard drives), 50ms might be a low estimate.

Actually I thought 50ms was a pretty high estimate, and that I was being conservative. Certainly, I would expect any delay to be dominated by disk seek time, which itself should never be more than the worst case of 10-15ms for a typical drive. I can't really see why it would be significantly slower than that, as all the other operations that would need to be completed should be around an order of magnitude quicker.

But the experience that people had with Vista SuperFetch is that it thrashed the disk constantly (which it shouldn't...once RAM is loaded, the disk should idle), [...]

No, the disk would pretty much always be active, as the "predictive" part kept "predicting" what would be best to (pre-)cache.

[...] and seemed to do stupid things like write out pages to the page file in favor of loading that RAM with "predictive cache". You can see how this would really slow things down.

I could, but I've seen little evidence to suggest it was actually happening.

But, even without this, a "predictive" cache should still put more weight to recently used pages from the active application, especially if there hasn't been any change of foreground app in a while (which is likely with a game). This would result in a user experience which is better by being more responsive, even if it's only skipping a few delays of 100ms.

An in-use foreground app should never be shuffled out of active RAM at all (assuming there's enough of it in the first place) - and, again, I've seen no evidence this actually happens, other than people on Slashdot (and other sites) divining what their computers were doing by watching the hard disk light flicker, or other similarly irrelevant metrics.

Re:So

[ls671]

Of course, if you want real time, any operation has to execute always in the same amount of time. I would suggest using a real time OS.

In the mean time, I remain convinced that setting up a swap space concurs with making the most efficient use of my resources, as it is probably the case for most users and most use cases. You can fine tune the swappiness level which is by default 60 on most distros. I never felt the need to change the 60 default value although.

Re:So

[siride]

This sounds like unnecessary optimization. On my machine, swap never gets touched, even when it is there, so what's the point of having it? Your point would be valid if, for my workloads, swap was getting used, but I still decided I'd rather have the extra disk space and got rid of it.

Re:So

[ls671]

It sounds like, given your use case, you have plenty of memory but limited hard disk space. If this is correct, I would guess you made the right decision ;-))

Re:So

[ls671]

I screwed up and replied to myself ;-))

http://slashdot.org/comments.pl?sid=1557492&cid=31290192

Re:So

[siride]

I have plenty of memory and also a good bit of hard-drive space. However, having a swap partition is a pain and is almost always wasted space eventually. If I really need swap, I can always create a swapfile.

Re:So

[bingoUV]

I don't know anything about Windows kernel internals, but:

The zeroing can be done entirely in software, don't you think? I describe my thinking in below points. Points 1 and 2 apply irrespective of whether there is a zeroing requirement or not.

1. When a process asks for a page (say 4kb) in memory, tell the process that you've got address n to n+4k. Mark in a kernel data structure, that this piece of memory belongs to this process.

2. Now, this kernel data structure is required frequently: whenever any process asks for data in any address, kernel needs to check if the process has the authority to read this address. So this kernel data structure is likely to remain in cache (L1/L2/L3).

3. Let us add a single bit in this kernel data structure for every page in memory, to indicate that this page has been newly allocated to a process and hence any data read request must return zero. (Since granularity of RAM is in bytes rather than bits, this bit can hide in a byte used for some other purpose within the same kernel data structure, so it may or may not increase the size of this kernel data structure.)This has 2 advantages:

a. Reading of fresh memory is super-fast. RAM bandwidth is of the order of magnitude of 10GB/s, or less. But for fresh memory, once kernel data structure's "freshness" bit is true, this data can be read in multiple TB/s speeds because all it has to do is return zero. A kernel equivalent of `cat /dev/zero`, if you will.

b. Second advantage, of course, is that no explicit zeroing of RAM is required.

Please let me know what you think of this.

30ms?

[Colin Smith]

Really? 30ms. Shit that's slow.

This IS RAM we're talking about here, right? y'know nanosecond stuff, 10^-9 not 10^-3 seconds.

Re:30ms?

[sopssa]

This is in the case you need to free like 1GB of normal RAM. Other than like Photoshop and games don't need that much. But anyway, it has to be zeroed out for security, and so that other apps cannot randomly get data from other apps.

Besides the technical points, 30ms to free that RAM is really low. You wont even notice that.

Re:30ms?

[Alwin Henseler]

This IS RAM we're talking about here, right? y'know nanosecond stuff, 10^-9 not 10^-3 seconds.

Yes, and nanoseconds (10^-9) multiplied by the number of memory locations to clear (10^6 when you're talking multi-MB chunks of memory) gets us right back in the millisecond (10^-3) range. Which is just a blink of the eye for us humans, btw.

Re:30ms?

[tdelaney]

This is another reason why having more than 2G of RAM on a 32-bit OS is dumb.

What are you smoking? Even on Windows 32-bit, without any modifications, the 2GB limit is per-application. I have yet to see a machine "lose" more than 1GB to address space (admittedly, no crossfire or SLI setups).

I have 4GB in my 32-bit development machine and regularly use it all. And I mean all - I use Gavotte's RAM Disk to create a 512MB RAM disk in the "unusable" portion that I use to hold my temp directory and other things I don't want to survive reboots. I've tried living in 2GB, and it's just not reasonable even with XP if you're forced to use Eclipse + JBoss ...

Re:30ms?

[ashridah]

You might want to grab a copy of Process Explorer sometime, and look at the stats it reports. you'll notice that windows actually spends idle time pre-zeroing ram, so that this is already done, in more than enough amounts. If your system is slammed, i could see having to pre-zero the pages, just before use, however, but it's not like it's not something that couldn't be done while waiting for other I/O operations to complete (since your system is slammed anyway :) )

My laptop currently has 2.8 million pages zeroed atm (it has 8gb, and I don't have much running right now, so there's not a lot to cache.)

Re:30ms?

[Anonymous Coward]

I guess you're never heard of PAE?

Re:30ms?

[SScorpio]

Have you tried using PAE in Windows. It was supported for the original release of XP, but Microsoft removed support for it in either SP1 or SP2. Their claim was buggy 3rd party device drivers that didn't handle support for PAE correctly so the system would become unstable. I'm not sure about Windows 7 but PAE does not work in Vista as I tried it with the original release and SP1. If you need 4GB or more just use Windows 7 64bit. Hardware manufactures have finally gotten their act together and you have stable 64bit drivers for things now.

Re:30ms?

[blueg3]

No, Windows actually uses PAE, and Datacenter versions of the OS support more than 4 GB RAM under PAE. However, almost all the versions of 32-bit Windows intentionally limit the physical address space size to 4 GB, even when PAE is enabled. This is apparently for driver compatibility reasons (so that physical memory pointers will never exceed 32 bits).

Re:30ms?

[Nutria]

Windows knows the differences between "free" RAM and cache RAM, and can't (can it??) be so stupid as to zero out what it knows to be cache.

Re:30ms?

[kc8apf]

That's why we have virtual address spaces. Each process gets a 4GB address space for a 32-bit OS. Only 2-3.5GB (depending on OS) will be available for actual program code (not shared libraries) and data. Of course, those virtual addresses get translated to physical addresses by the OS on a per-page (or at least per-range) basis, so all 4GB of physical RAM can be used assuming there is more than one process running.

That's all a simplification of how modern VM subsystems work, of course.

As to memory "stolen" by hardware, the big one there is video cards. A fair number of OSes still map the entire video memory into the physical memory address map. You might think this would be a problem, but on the x86 architecture at least, physical addresses have been 36 bits for a while. Even with a few video card with 1GB of video ram each, there is still enough space to address all 4GB of physical RAM.

Re:30ms?

[commodore64_love]

Dumb Question (perhaps):

Exactly how aggressive are Windows Vista and Vista 6.1 (Seven)? If I was a billionaire with more money than sense, and filled my PC with 1000 gigabytes of RAM, would Windows 7 systematically copy over my whole hard drive to the RAM? (Talk about super-slow boot time.) ;-)

Also what's the minimum RAM that Seven's aggressive caching expects to see before it falls-over on itself? 1/2 GB? 1/4 GB? 1/8th GB (128 MB)? Just curious.

*
* This message typed on a Commodore Amiga with 1/2 MB - for fun.

Re:30ms?

[siride]

Wrong, because you have a 32-bit physical address space which is global, not per-application. Unless you use PAE, you can use at most 4 GB of RAM with a 32-bit OS, and a lot of that gets eaten up by device and BIOS mappings and holes. cat /proc/iomem to see what I mean.

Re:30ms?

[ashridah]

I didn't make a snide remark. I pointed out that windows maintains a pool of zeroed ram, and will constantly fill it when it's able to.

Windows will allocate from that pool first, and then top it back up when it hits a threshold of low free memory from supercache.
It then has the luxury of handling this in the background, so you're not waiting for the system to zero memory out, unless you somehow manage to totally max out the memory you've got in use.
It's at this point when you've got bigger problems, like the fact that the disk will probably become your bottleneck, not the speed at which you can zero out ram.

I didn't miss your point, I'm suggesting that you're not thinking like an operating system designer, and saying "how can i shorten the critical path of giving a process the memory it requests?" The answer is: Pre-zero memory, and fill the zeroed pool from the cache as needed when you get low. There's no "wait" for zeroed out memory, at this point.

Re:30ms?

[ashridah]

You'd probably have to read something like Windows Internals for an answer to that.

The answer is, it's probably designed to scale, or turn a lot of its behavior off or ramp up under lots of different, interacting, conditions.

Doing this kind of thing to ensure performance is reasonable on low memory conditions isn't really a large problem, and probably isn't something they didn't think of.

Re:30ms?

[shutdown -p now]

Rumor it is that one of the reasons why 7 feels so much snappier than Vista is because superfetch was tweaked to not be overly aggressive in caching things - 7 still caches a lot, but it's much less gung-ho about when it feels an application is suitable for that.

Re:30ms?

[tdelaney]

And what relevance does that have to the OP's statement that "having more than 2G of RAM on a 32-bit OS is dumb"? BTW, PAE is used by default from XP SP2 (I think - it's required for DEP). How do you think I use my full 4GB with 32-bit Windows? Gavotte's RAM disk uses PAE to access the "lost" memory (several other RAM disks do the same thing).

Re:30ms?

[rtb61]

I can guess that over aggressive caching would be targeted at improving only one companies products at the expense of other companies products. Pre-fetching programs that you might use is really just a marketing fudge to make certain programs look like they perform better than they actually do, with the added benefit that programs outside the queue now wait while something else is loaded then unloaded prior their getting loaded.

Most used programs will generally get loaded immediately following boot, this is after fighting over windows delayed services booting marketing fudge to get a partially functional desktop to appear as soon as possible. So they you now add pre fetching of guesses of which program might be loaded but not looking at when it might be loaded.

Re:30ms?

[caywen]

Not to mention that zeroing memory out strengthens security (if only a bit). Fewer passwords and secrets sitting around waiting to be paged out to disk.

Re:30ms?

[FooBarWidget]

Then what *is* the point? That caching disk data in RAM is stupid? Pretty much every modern OS does this, and has done it for years. Linux, OS X, FreeBSD, etc. all do this. If you don't like this then I guess you should use DOS.

Re:30ms?

[erayd]

...unless you use PAE, you can use at most 4 GB of RAM with a 32-bit OS, and a lot of that gets eaten up by device and BIOS mappings and holes.

That's utter garbage. 32bit operating systems have been addressing considerably more than 4GB of RAM for many years using PAE - the maximum address space using PAE is 64GB.

See here for a more detailed explanation.

Re:30ms?

[siride]

Clearly you cannot read. Did I not say "unless you use PAE"?

2GB on XP isnt enough anymore

[Hadlock]

I think it's just a sign of the times. I regularly bump up against my 2GB ram limit (once a day) if I have GIMP/Photoshop open, 3 or 4 Chrome windows open with 10-20 tabs each (many of those being youtube videos), usually a videogame in the background (Windowed No Border mode at full or almost full screen resolution rules), along with whatever else I'm doing, a paused VLC video, steam, and any other background apps + whatever I'm working on currently. This isn't a problem in Win7, it's a problem of Leaving a Bunch of Shit open all the time.

Re:2GB on XP isnt enough anymore

[Courageous]

*shrug*

Windows 7, 64 bit here. 8GB RAM. Intel X25-M SSD. Seems like nothing I do can make my computer "sluggish".

C//

Re:2GB on XP isnt enough anymore

[Hadlock]

I haven't had any issues with sluggishness, it's just that Chrome tends to die when I hit the 2GB cap. It's speedy as all get out up until that point.

Re:2GB on XP isnt enough anymore

[ArchieBunker]

Just curious, whats the point of having so many open tabs with youtube videos? Do you keep having to refer back to them?

Re:2GB on XP isnt enough anymore

[grimJester]

Windows 7, 64 bit here. 8GB RAM. Intel X25-M SSD. Seems like nothing I do can make my computer "sluggish".

Same here, down to the SSD. I (obviously) still have only 60MB "free" memory but 6240-6250MB available.

Re:2GB on XP isnt enough anymore

[negRo_slim]

I often find myself with a long list of tabs open presenting a history of my travels during searches allowing me quick backtracking to various points. Youtube or other flash content isn't unusual in many tabs. It's just easier than closing this or that tab only to find the one you closed had a potential link or piece of information you now need. When all is said and done I just X the Window and start fresh, but up till that point you could potentially have a vast number of pages of all sorts of content open at once without much thought.

Re:2GB on XP isnt enough anymore

[Hadlock]

Sometimes it's a catchy song I want to listen to again later, but probably don't want to favorite. Or I did a search and found more than one interesting, tangential video I might want to watch later, but don't have time to now. Other times I simply forget to close them. Sometimes I leave them open to link to later in a blog or email/facebook etc. Maybe if there was some sort of intermediate between "youtube favorites" and "web browser history", I would replace my current system.

Re:2GB on XP isnt enough anymore

[ls671]

> I regularly bump up against my 2GB ram limit

I ALWAYS bump up against my 4GB ram limit and it is perfectly normal ! ;-)))

The shit you "leave open" will be swapped out, so there is no problem there either....

I currently have 355 processes running on my system; 51 bash shells, 33 httpd, 65 rotatelogs, 3 XVNC server, 3 VMWare hosts, etc...

And it all runs as smooth as a baby as long as you do not try to use ALL the processes at once ;-))

http://slashdot.org/comments.pl?sid=1557492&cid=31212938&art_pos=2

http://slashdot.org/comments.pl?sid=1557492&cid=31213022&art_pos=1

http://slashdot.org/comments.pl?sid=1557492&cid=31213096

Re:2GB on XP isnt enough anymore

[jon3k]

I highly HIGHLY recommend a flash blocking add-on like FlashBlock for Firefox. There will be a play button where all the embeded flash videos would be and it won't load them until you click play. You can of course whitelist sites that you'd like to load all flash from. But now you don't have to have those 10 pages in tabs each with 2, 3 or more flash ads or graphics eating up CPU cycles.

Re:2GB on XP isnt enough anymore

[antdude]

2 GB of my old, XP Pro. SP3 is enough. I haven't seen my HDD swap like crazy due to lack of free memory. It's fine and I am a heavy mutltitasker (e.g., SeaMonkey v2.0.3 hogging 250 MB of RAM!, Trillain v4 Astra, SecureCRT/PuTTY, Winamp, video players, Office 2000).

Re:2GB on XP isnt enough anymore

[Nutria]

FireFox has excellent History Search.

Re:2GB on XP isnt enough anymore

[cerberusss]

*shrug*

Windows 7, 64 bit here. 8GB RAM. Intel X25-M SSD. Seems like nothing I do can make my computer "sluggish".

C//

Car analogy follows.

*Shrug* Ford F-450 SD Crew Cab here. Reinforced deer catcher. 4x4 wheel drive. Seems like not a single pedestrian can slow me down.

Re:2GB on XP isnt enough anymore

[Courageous]

I got modded funny, which was strange. Here you go, you can have my funny mod points. :-)

Seriously though, this wasn't that much of a vanity purchase. The CPU and GPU are modest. RAM these days is CHEAP. I did spend a good $250 on the SSD, but most of my storage is cheap SATA. It all mixes together quite well. That SSD makes my system feel sooooo snappy. It doesn't hurt that Windows takes less time to boot than the BIOS does to POST.

C//

Re:2GB on XP isnt enough anymore

[cerberusss]

Yeah, the SSD bucks are well spent. The stuff is outperforming normal disks on almost every front and the disk sizes are big enough to hold most data. And I've got to agree on the RAM, especially now since Windows will properly gobble it all up and spend it on disk buffering if you don't use it for apps.

We live in great times :-)

Re:2GB on XP isnt enough anymore

[Hadlock]

There's no flashblock plugin for Chrome. Welcome to the 2010's, dude! Firefox is soooo 2000s. Plus I'm not going to put up with Mozilla's ideological Ogg vs. h.264 BS anymore.

Re:2GB on XP isnt enough anymore

[bingoUV]

I feel Chrome is behind the times. Firefox has good solutions for these problems:

1. Tree-style tabs: Maintain related tabs in a group, with tree-like relationships between different tabs. When done with a subject, close the whole tree related to that subject. I use Tab Kit, which provides some other functionality in addition to tree-style.

2. Temporary bookmarks: For bookmarks that are of interest now but may not be in a few weeks

Re:If only the cache were actually -good-

[ClosedSource]

Sure, I know if my compiles seem to be slow there's nothing better for improving my productivity than diving into the cache.

In any case, if you want to compare OS cache performance you might at least try to use the same compiler and the same language on both machines.

Re:If only the cache were actually -good-

[Runefox]

As I understand it, it doesn't cache data, it caches applications (and I think also fonts and other often-used things). So startup time for your web browser, e-mail client, IM client, and any other applications you use often will be much faster. For example, Google Chrome loads almost instantaneously on my system, from a cold start. It won't keep a memory cache of things that applications do, and hence it won't speed up compilation, rendering, etc.

For that matter, 2GB of RAM isn't a whole lot to cache with when you think about it. The sweet spot for Vista/Win7 is really around 4GB or higher for SuperFetch to really shine.

Re:If only the cache were actually -good-

[zippthorne]

How would you do either of those activities with Linux. I know you can cat stuff into the cache, and you can control the "swappiness" and something about dirty inodes in proc/sys, but how could you say "I want this file to be in the cache no matter what" or "what files are in the cache" other than by creating a ram disk and putting them there (and then wondering why your ramdisk files are also cached because you used /dev/ram instead of tmpfs...) manually?

Re:he's a native speaker

[sopssa]

This is completely offtopic, but you seem to come with this message in every thread so I answer you one time here - I am from sweden, not uk.

Been using 7 awhile now

[mschuyler]

Seems to use 1.3 gig no matter what I do. It's got 4 gig total. Boots in less than 2 minutes. It makes mistakes a lot faster than the old machine, which was just shy of a brick when I switched.

Re:Been using 7 awhile now

[jon3k]

I agree, 2 minutes is very slow.

Re:It's a matter of definitions

[peachstealingmonkeys]

I see what you did there. However, the alarming claim was misleading. W7 has a dynamic memory utilization, that's pretty much it.

Re:he's a native speaker

[zach_the_lizard]

And though your English has some minor mistakes every now and then, it's not like you are writing in such a way that we cannot understand you. Your English is better than my grasp of Spanish by many miles. Congrats.

Re:It's a matter of definitions

[Beelzebud]

It's not "hogging" memory if it dumps it the second you start up a program that needs it... It also doesn't make your system "appear" faster, it makes it faster. I paid for all that RAM, I don't mind it being taken advantage of; that's why it's there in the first place...

Same compiler on both machines

[tepples]

In any case, if you want to compare OS cache performance you might at least try to use the same compiler and the same language on both machines.

Use Visual C++ on both machines? Good luck getting it to run in Wine.

Use GCC on both machines? The last time I tried GCC on Windows, it produced bloated binaries for any C++ program that uses <iostream>. A quarter megabyte for Hello World, on two platforms (MinGW targeting Windows/x86 and devkitARM targeting GBA/ARM7)? Give me a break.

Re:Same compiler on both machines

[ClosedSource]

Well, if you can't do an Apples vs. Apples comparison then don't. There's no point in performing a comparison that you know is flawed from the start.

Re:Same compiler on both machines

[tepples]

Bloated C++ binaries were caused by GCC having only a static c++ runtime on windows

And it still has only a static C++ runtime on, say, Nintendo DS because there's no libstdc++.dll in the BIOS.

Newer MinGW GCC releases (4.4 series) include libstdc++.dll

Which of course I'd have to distribute with each copy of each program because the user doesn't already have it. Even so, how many bytes does hello world take under newer MinGW? (I don't have Windows in front of me at the moment.)

Re:Same compiler on both machines

[tepples]

Cygwin is copylefted. This is fine if you don't distribute your apps or if you make money on services. But I haven't seen a way to make money from, say, a copylefted single-player video game.

Re:Same compiler on both machines

[tepples]

Almost all software for Windows is compiled with Visual C++. Almost all that software runs under Wine. Whether it runs under Wine or not has NOTHING to do with Visual C++.

Using the same compiler on a Windows box and a Linux box, as ClosedSource suggested, would require running Visual C++ on Wine.

ALL Visual C++ programs require the "bloatware" dll.

This DLL comes with Windows. The DLL used by C++ programs compiled by recent MinGW versions does not come with Windows.

Re:Same compiler on both machines

[theArtificial]

Neat blog - I've learned a lot.

Linux does that

[Animats]

Linux uses available memory for cache, and rather aggressively. All available memory can be filled with cached file blocks. This happens routinely on systems which have big randomly-accessed files open, like databases.

There's nothing wrong with this, except that, once in a while, Linux hits a race condition in prune_one_dentry, causing an "oops" crash, when there's an unblockable need for a memory page and something is locking the file block cache.

This is one of the Great Unsolved Mysteries of Linux. Linus wrote about it in 2001 ("I'll try to think about it some more, but I'd love to have more reports to go on to try to find a pattern.. "). As of 2009, this area is still giving trouble. The locking in this area is very complex.

Re:Linux does that

[Angeliqe]

Linux uses available memory for cache, and rather aggressively.

That explains some things. I used to run top after a system had been running for awhile and always wondered why my memory usage was so high compared to when the machine first started up. It seemed that even when I closed a program, the memory was not reported as being "free" memory. However, if I loaded one of my memory hogging 3D games in wine and then shut that off, the memory usage would drop drastically.

Re:Linux does that

[Simetrical]

Linux uses available memory for cache, and rather aggressively. All available memory can be filled with cached file blocks. This happens routinely on systems which have big randomly-accessed files open, like databases.

Actually, databases will typically maintain their own cache, bypassing the OS page cache. A notable exception is MyISAM, the default storage engine for MySQL: that uses the OS page cache for data (although maintains its own cache for keys).

In practice, my Linux machines always have all memory filled within an hour or two. You just have to read or write a gigabyte or two of non-temporary files.

Re:Byte vs. megabyte

[Colin Smith]

If one byte takes 10^-9 s, a million bytes take 10^3 s.

Only if you do them one at a time, one after another, waiting till each had been cleared. That'd be... Wait this is Microsoft we're talking about here... Carry on.

 

Re:It's a matter of definitions

[Superdarion]

Windows 7 is sucking up your system memory to make Windows appear faster.

So windows has a feature which makes everything run faster and yet it only "makes Windows appear faster", instead of making it actually faster?

It seems to me that windows is only using hardware in a rather intelligent way: if it's not being otherwise used or needed, it uses it to boost performance.

Linux does the same thing, as far as I know, and you don't see anybody calling Linux a memory hogging OS.

Tits on a bull

[RobbieCrash]

Why does anyone care about having piles and piles of free RAM? What use is empty RAM? Runway behind and Sky above, nice that it exists, but totally useless. I wish Linux prefetched stuff. If I open program 1, and ALWAYS open program 2 after, why shouldn't the OS preload program 2?

As has been pointed out, prefetched data is dropped if the RAM it occupies is needed in an infinitesimal amount of time, is your life so short, or your tasks so time sensitive that 10 or 15 ms is really a big deal?

Re:Tits on a bull

[MichaelSmith]

I wish Linux prefetched stuff. If I open program 1, and ALWAYS open program 2 after, why shouldn't the OS preload program 2?

Wasn't this the issue with ext4? It keeps a huge cache in memory so if you write to a file and don't ffush it correctly you can lose your content on power down.

(off to put fflushes before close in all my programs)

Re:Tits on a bull

[RobbieCrash]

That makes no sense. Superfetch is run at the lowest priority, all other reads/writes interrupt anything it's doing and it only uses idle processor time.

At boot, Windows is doing other things not prefetching. Turn superfetch on and monitor your active processes at boot, you'll see that the superfetch PID is sittng all the way at the bottom of everything. Its IO is negligible as is its CPU usage until AFTER everything else has loaded and your system has gone back to twiddling its bits for a few hundred thousand processor cycles.

Re:Tits on a bull

[mrnobo1024]

fflush() just flushes stdio's buffer, so that any data written to the file is sent to the operating system (via write() on *nix), making it visible to other programs. It does not flush anything in the operating system's own buffers/caches. Also, fclose() calls fflush() automatically, so calling both is redundant.

You're probably thinking of fsync(), a system call that does actually force data to disk. And should almost never be used, unless you enjoy waiting through several seconds of disk grinding and general unresponsiveness.

Re:Tits on a bull

[MichaelSmith]

Yeah that would be it. I recall that some people were suggesting during the ext4 articles of a couple of years ago that you have to use fsync() on ext4 to be sure your data is on disk.

Re:Tits on a bull

[tftp]

That makes no sense. Superfetch is run at the lowest priority

It's not the CPU that is a bottleneck, it's the HDD. When a large application, like FF, starts it needs to load tons of DLLs, read lots of files. It can't possibly help that some other reads are interleaved into that process; and once the HDD is executing the SuperFetch request it won't be available to execute FF's request. The SuperFetch service needs a limit on HDD use - say, one HDD access per second, and only if there was no other access within last second. But it doesn't appear to have any such limit, the only constraint is the priority of the thread, and with modern CPUs it doesn't do a thing, resulting in the HDD thrashing like mad for minutes.

Re:Tits on a bull

[tftp]

You can't prioritize I/O down to the hardware, and that's the problem. Once the heads start moving the HDD is committed to get them on a certain cylinder, even if the CPU has now a higher priority request.

Similarly, once the reading of a certain number of sectors starts it will be completed by the HDD, though it takes time (one rotation of platters.) If SF needs to read an adjacent track it won't be allowed to, due to priority, but then the HDD will be seeking back and forth (thrashing.)

On top of that, SuperFetch reduces efficiency of the cache that is inside the HDD - it gets diluted by data that is not going to be accessed any time soon.

Re:Tits on a bull

[cbhacking]

Odd... your experience is the exact opposite of mine. I find that SuperFetch, *especially* on a laptop, is essential to speeding things up specifically because I don't have to wait for the data to be read off the disk before I can run a program. Startup doesn't seem any slower to me (I'll grant you that I haven't timed it) and unlike XP (no SuperFetch), either Vista or Win7 (both with SF enabled) is usable immediately after logging in. Sure it's slower while background stuff loads, but it's not like the computer is nearly frozen for a while, the way XP used to do. Once bootup is past, SuperFetch becomes essential to remove the bottleneck of that really slow laptop hard drive when I try to run something.

Re:Tits on a bull

[thetoadwarrior]

Depends on the circumstances. If I'm just browsing the net then yes I expect to be use little memory. If I'm playing a new PC game then I plan on using every last ounce of memory.

Because there some tasks that I do want to use as much memory as possible then I would naturally want the OS to use as little as possible. I don't want to have worry about stupid little programs running in the background.

Instead of executing even more programs to cache crap to run a little faster, PC retailers need to start selling better PCs without crapware installed on top and Microsoft needs to make their OS leaner. Those two things will achieve better results than SuperFetch. MS does these things to help people because most people don't know how to sort out their new PC and so MS is just patching over the underlying problem.

Memory shouldn't always set there unused but all of it doesn't need to be used all the time.

Re:Tits on a bull

[seeker6182000]

Vista and Win7 has I/O priorities, but it is done in the I/O stack above the hardware. Low priority IO that is not in flight on the hardware will be delayed. The longest a single low priority I/O operation can block access to the hardware is the length that it takes to complete, which is relative short, less than 10 ms typically. Once higher priority I/O requests are received, low priority I/O is delayed until all higher priority I/O is completed. After no higher priority I/O requests are received for a significant period, then low priority I/O resume. SuperFetch has an extremely small interference with foreground application I/O in Win7 due to this mechanism. In addition, if FF is a process you use frequently, SuperFetch will have the relevent code and data files in memory before you launch it. The disk track buffer is used mostly for disk writes, since the I/O bandwith to the rotational media can't keep up with the bus transfer bandwidth.

Re:Tits on a bull

[RobbieCrash]

You're wrong, and need to read how SuperFetch actually works.

As for the drive's internal cache being diluted with data that isn't going to be used, you are going to be sad to know, hard drive caches are used for prefetching data. Yes they contain read/write info and the command queue, but they also prefetch raw bits based on predictive algorithms.

Re:Tits on a bull

[RobbieCrash]

What do you want your memory to be free for? So you can brag about how much wasted RAM you have? It's like having a house filled with closets that go empty and keeping all your clothing in a storage yard across town.

Re:Tits on a bull

[Eskarel]

Sort of.

The issue with ext4 had to do with dirty cache not being written to disk in a timely manner. This meant that you could have disk writes you thought had already happened and certain actions could get out of sync because of they way they'd been written.

The cache in this case is not dirty(in fact it's read only) so there's no risk of losing data (since most of the time the data on the disk and the data in memory are identical and when they're not the disk is right).

Re:Byte vs. megabyte

[Anpheus]

Everyone knows there's a shadow-bus on the motherboard that only open source operating systems have access to.

Re:It's a matter of definitions

[Subliminalbits]

Describing caching as a way Windows makes your computer "appear" faster is really a little disingenuous. If that is the only metric for your complaint then you should be angry that your processor caches as well. After all, your processor takes the time to check two or three caches every time it issues a move instruction. If it misses every time, then it has to pick what to throw out of the cache and read directly from memory. Wouldn't it be so much better if it just made a fetch to ram every time there was a move instruction? After all, your processors caches only "appear" to make your processor faster right? The question that people should be asking if they want to get upset about SuperFetch is does this approach to ram use benefit the user enough to be worth the extra complexity in the operating system's memory allocator.

Re:It's a matter of definitions

[nedlohs]

Yeah much better to not use the available free memory and just keep loading the same things off the slow disk over and over again.

That way you can always 2GB of free RAM!

And it doesn't make Windows "appear faster", it makes it faster. Caching is one of the oldest performance optimizations in computing. Everything uses it.

The linux box I have here would drive you crazy:

# head -4 /proc/meminfo
MemTotal: 8156032 kB
MemFree: 51000 kB
Buffers: 59504 kB
Cached: 7584944 kB

Re:he's a native speaker

[JustOK]

People always mix up Sweden and Swindon

Re:If only the cache were actually -good-

[ls671]

Programs and compilers have nothing to do with disk read/write caches, this is handled at the OS level.

Programs ask to read/write from/to disk and they have no idea where the OS get the data from.

You and the GP might be confusing CPU cache that has its own dedicated memory and disk cache that uses the OS RAM.

Re:If only the cache were actually -good-

[ClosedSource]

My intent was not to suggest the proper way to test cache performance. I was merely stating that any comparison experiment should hold all other variables constant.

This is one of those great mysteries of life

[davmoo]

if everyone is so afraid of their computer memory being used to the fullest, why do these people install so much of it?

I've got 8GB of ram in the machine I'm on at the moment, and I want the OS and applications to use it to the fullest and most efficient extent possible at all times. I didn't install a 64-bit OS and 8GB of ram so that I can see 6GB free at all times.

Re:This is one of those great mysteries of life

[Overzeetop]

if everyone is so afraid of their computer memory being used to the fullest, why do these people install so much of it?

Most users remember back to at least the 90s. You had to install enough ram to do what you needed (load the OS and program(s) - WinNT had the audacity to require 8MB to run well). There was no caching of any useful sort, so your free memory was really a measure of how many programs you could load. Programs, like Photoshop, added scratch files to overcome the physical RAM limits, but at a horrible performance penalty should have to actually use it. "Free RAM" became synonymous with "how many things you could do or open simultaneously."

All modern operating systems have moved on, but people haven't been educated about this. They remember how bad it was when they ran out of memory, and panic when the OS reports it's almost full. Honestly, it would be far better if MS would have reported the cached memory differently. I don't really care how much memory is used as superfetch cache most of the time - I'm more concerned with the total active usage. My netbook "only" has 2GB, but I do some 24/96 audio recording with it, and will occasionally work with photoshop images, so I am concerned if I have less than 500-600MB free when I open a session as I'm likely to exceed the physical RAM. I can read the data, so it's not a big deal, but others freak out about it.

Re:This is one of those great mysteries of life

[nine-times]

But it makes me feel so awesome to see 6GB free. I'm all like, "Damn, I have a lot of RAM!" When the RAM is all fully, my system monitoring graphs don't look at cool. I also like seeing my CPU utilization showing 4 cores, each idling around 1%, and having a multiple terabytes of free space on my hard drives. All those graphs get ruined if you actually use your computer for stuff.

Re:This is one of those great mysteries of life

[davidbofinger]

Dude, that's not the way to think. You've got to get in the habit that when you look at your system monitor and see "7.4GB used" you think, "Wow, I am such a power user. This $hit hot machine is just barely keeping up with my requirements." If that doesn't make you feel good then you're beyond help.

Secondary problem - spinning up HDD

[invalid-access]

It seems a secondary problem using SuperFetch, is that the HDD is woken up, or prevented from spinning-down when it otherwise could have been. ie. Bad notebook/netbook battery life. Anyone with insight into this?

Re:Secondary problem - spinning up HDD

[Anonymous Coward]

There's so much s**t going on in the background on every Windows machine that the HDD will never spin down even with SuperFetch disabled. Practically every program writes to the registry every few minutes...

Re:It's a matter of definitions

[Suiggy]

If Windows 7 actually uses that much memory it's not scaremongering, it's memory hogging. Whether it's using it on not is a pretty fine distinction, it's still using it just because it can. If something else needs it, Windows has to decide if it wants to let go of it or not.

So are you saying Linux, BSD, Mac OS X and pretty much every other modern desktop OS other than Windows XP are also memory hogs as well? Because they also do the exact same thing and use up all of the free memory for caching, marking it as available.

Too much?

[beej]

If Windows refused to use your RAM that you had installed, now that would be an issue. But fully using RAM? This, on its own, is not something to complain about.

Re:Too much?

[mgblst]

Wow, this article has bought out all kinds of stupid.

People don't want it to use all of their ram, when they have no apps open, do they?? They don't mind when they are running PS, Firefox, etc... if all there ram is used. But when they haven't started anything yet, it can be a little confusing.

Re:Too much?

[beej]

Wow, this article has bought out all kinds of stupid.

People don't want it to use all of their ram, when they have no apps open, do they??

Well, it boils down to a simple question: do you want this app to launch now, or do you want to wait for it to load off disk? That's really the crux of it.

But when they haven't started anything yet, it can be a little confusing.

We're lucky there's not a meter telling them that their CPU cache is full all the time, too, I suppose.

You're missing the point

[jon3k]

The last article specifically said RAM was nearly exhausted and there was excessive paging to disk. No one cares if RAM is full or not, if it's unused it's wasted anyway. The concern is having 85% memory utilization and then paging memory out to the pagefile.

Re:You're missing the point

[mystikkman]

YOU are missing the point. The last article seemed to be guessing that excessive paging was occurring *because* RAM was full. It didn't specifically state paging was happening. Any ambiguity about that has been resolved by this article...

So it was little surprise that, upon checking my reported stats on XPnet, I found that I too was in the "alarming" position of having virtually no free memory.

He had 1.3 Gigs of memory. I hope you and other anti-MS zealot posters in the earlier article who were claiming that the article specifically said there was excessive paging eat crow now.

Re:he's a native speaker

[tdelaney]

Or perhaps he was comparing proficiency at second languages ... (english as a second language vs spanish as a second language).

that was supposition

[YesIAmAScript]

The only data in the article you refer to was captured data from XPnet that said that >90% of RAM was in use in Windows 7 machines. There was no data saying it led to swapping, that was supposition.

http://www.computerworld.com/s/article/9158258/Most_Windows_7_PCs_max_out_memory

Re:that was supposition

[TimothyDavis]

Additionally, the memory manager will write active paged memory to disk without paging the data out of RAM. This is done so that when you hibernate your machine, much of what is in memory is already written to the disk. The machine can therefore hibernate much more quickly than if it had to write the full active working set of RAM to disk.

Re:If only the cache were actually -good-

[Anonymous Coward]

I think this is what redhat called their ReadAhead Manager (or something of the sort). I believe Ubuntu has or had something similar.

While it sounds like a good idea, it doesn't really pan out, it just makes your boot time longer. Who cares if you improve the speed of opening your browser when the rest of the OS finally loads. Lots of people only open their browser once each time they boot up anyways.

I'm not much of a fan of binary preloading, but the OS/caching memory model in linux does actually give you a huge and visible performance gain. This can be easily seen by trying to autocomplete all possible programs in bash under linux, twice, then comparing the same thing in MSYS (mingw32's linux environment) with bash.

*Spoiler* the second time you try and autocomplete the list it will be extremely fast.

  This same effect speeds up directory listings, Open/Save dialog boxes (duh!), Traversing paths, the loading of frequently used binaries and libraries, and possibly the visualization of thumbnails in nautilus with image directories.

In theory

[Dunbal]

A good OS uses all the RAM, and allocates available free blocks of RAM to the programs as required.

However using the greater part of a gigabyte plus paging to the hard drive just to display the desktop and run the low level functions is inexcusable and points to either a) memory leak b) the OS is doing something legitimate you are unaware of, like indexing files, etc c) the OS is doing something illegitimate like sending the contents of your hard drive to someone in Redmond, the NSA/FBI or the RIAA/MPAA or d) your system has been compromised by a virus. I can't really think of any other possibility.

Re:In theory

[jpmorgan]

Or e) you're using an operating system which learns your common usage habits, and preloads applications and data it predicts you'll probably want into otherwise unused memory

Duh?

[Vyse of Arcadia]

Isn't this how RAM is supposed to utilized anyway? I don't want my OS flushing out RAM if I'm just going to use the same data later on.

Re:It's a matter of definitions

[ls671]

I have encountered people on /. that would like to put memory chips in their machine without having the OS touch it. We need to write a FAQ about this. ;-)

http://tech.slashdot.org/comments.pl?sid=1557492&cid=31213314

Re:If only the cache were actually -good-

[jon3k]

There used to be a ramdrive.sys but turns out the OS is smarter than us when trying to use RAM as a cache. Here's a reference to it in an article for troubleshooting memory problems: http://support.microsoft.com/kb/142546

Windows 7 and Vista Lie about memory usage

[Rashkae]

Both articles miss some very big and important points. Back in the day of Windows 2000 and XP, the Task Manager chart reported the memory comit charge. Basically, that was the amount of memory applications (and Windows) requested allocated. This does not mean that much memory was actually used, but with the exception of very badly written/buggy programs, it should be close. As a rule of thumb, if you look at that and see that your commit is significantly larger than your RAM, you know you're probably in trouble and will be very reliant on swap.

Windows Vista and 7 report something completely different. The chart shows ram memory used minus cache, an almost useless metric, but it does not indicate how much 'total' memory, real and virtual, is allocated. If you look at the screenshot in the ars aritcle, you will see that the commit charge is over 3GB. That's a lot of memory, and doesn't include cache!.

At the end of the day, however, a bare bones Windows XP would require about 120MB of memory, whereas Windows 7 is around 1GB. That sounds like a big difference, but we are talking several years of new features and eye candy. Ultimately, when you drill it down, it means that Windows 7 requires $20 more worth of memory. An insignificant issue, so long as you keep that in mind when designing a system for Vista / Windows 7. (ie, make sure that any computer or device destined for those OS's have at least 2GB of ram)

Re:Windows 7 and Vista Lie about memory usage

[m_pll]

Back in the day of Windows 2000 and XP, the Task Manager chart reported the memory comit charge. Basically, that was the amount of memory applications (and Windows) requested allocated. This does not mean that much memory was actually used, but with the exception of very badly written/buggy programs, it should be close

Not necessarily. Many programs commit large chunks of memory in case they need it later but only use a small portion initially. This simplifies program logic because you don't have to free and reallocate the buffer when you need more space, deal with potential reallocation failures etc. Or a program might want to specify a larger-than-default stack commit size to make sure it doesn't hit a stack overflow if it tries to extend the stack while the system is temporarily out of commit (most services and other system critical processes do that). Or it might map a copy-on-write view of a file, in which case commit is charged for the entire view but no extra physical memory is used until the program actually writes to the pages. Etc etc... The end result of this is that you can't really say anything conclusive about physical memory usage by looking at commit charge

Commit charge is a virtual memory metric. It's great for detecting memory leaks and deciding how big your pagefile needs to be, but not so great for understanding physical memory usage. Often it might seem like there is a correlation between commit charge and physical memory, but you can also find systems that are very low on available RAM yet have plenty of available commit, and vice versa.

Task manager now shows used physical memory (defined as Total - Available). Available memory is the most straightforward way to understand whether your system needs more memory or not, and this is why in Vista/Win7 it was chosen as the main indicator of "memory usage".

Re:Windows 7 and Vista Lie about memory usage

[Rashkae]

Still fails badly though, because the task manager will show lots of available memory when lot of caching is being done, depending on how the system is tuned. (sometimes known as swappiness in Linux, I'm not entirely sure where to find the tuning parameters in Windows). It's not very hard to find systems that consider more than 30% memory available, but are considerably slowed down by swap activity. Of course, the only way to really prove that is with some kind of swap monitor that looks for excessive swap usage (in particular, watch for systems that need to swap out at the same time as they swap in.) But without keeping a hawk eye on minute by minute page faults, I always find the commit charge to be a very reliable eyeballs estimate. If you have a system with 2 GB of memory and 3GB of commit, it's almost always a given you will soon suffer from memory starvation.

Re:Windows 7 and Vista Lie about memory usage

[Eskarel]

That's why you have metrics for page faults. If you don't have any page faults you're not swapping. If you have some you're probably slightly over utilized or doing something odd. If you have a lot, you have a problem.

Re:Windows 7 and Vista Lie about memory usage

[m_pll]

Still fails badly though, because the task manager will show lots of available memory when lot of caching is being done

And what's the problem with this? If memory is shown as available then it *is* available. When an application asks for a page of memory, and there are no free/zeroed pages left, the memory manager will take a "cached available" (a.k.a "standby") page, zero it and give it to the application.

It's not very hard to find systems that consider more than 30% memory available, but are considerably slowed down by swap activity.

It's also easy to find systems that are paging like crazy while their commit charge is nowhere near the total amount of RAM. Any workload that involves a lot of cached or memory mapped file IO can potentially cause this, because file backed pages don't consume commit but they can definitely consume a lot of RAM.

The situation you describe (lots of paging even though 30% of RAM is available) is actually not all that common. Let's say we're talking about a 2 GB system, so 30% is around 600 MB of available pages. This is far above the threshold at which the memory manager starts actively trimming working sets. That makes it pretty hard to come up with a scenario that would cause a lot of paging while maintaining at least 600 MB of available memory. Consider the steps that need to occur before a page can be hard faulted on:

1. A page is allocated and added to the working set of a process
2. The page is trimmed and placed on the modified list if it's dirty (otherwise it goes directly to standby list)
3. The page is written to the pagefile (or its backing file, if it's a file page) and placed on the standby list
4. The page is repurposed from the standby list
5. Finally, the page is accessed again, causing a hard fault

The first hurdle is step #2. If there is no trimming, pages will simply accumulate in process working sets (until available memory drops below the trimming threshold). And even if the app explicitly trims its pages, they will normally sit on the standby list for a long time. The standby list is FIFO, so somebody will need to push 600 MB worth of pages into it before a given page is repurposed.

Low available memory on the other hand is a great predictor of paging. If available pages get low that means the standby list is much shorter, and trimming is more frequent. Both of these are necessary conditions for paging. Also note that it doesn't matter whether RAM is being consumed by pagefile-backed or file-backed pages - available memory will drop in either case, unlike commit charge which only counts pagefile-backed pages.

If you have a system with 2 GB of memory and 3GB of commit, it's almost always a given you will soon suffer from memory starvation.

I'm typing this on a 2 GB x64 win7 machine. It's running two Office apps, Visual Studio, several browser windows and a couple of memory hogging apps that we use internally where I work. Commit charge is stable around 3.4 GB, but there is ~800 MB of available pages. Reads from the pagefile (according to resmon) are very sporadic - maybe a couple per minute. Pagefile writes are even less frequent (Page Writes/sec in perfmon shows a steady 0, and !vm 9 in the kernel debugger shows that the last write occurred 12 minutes ago).

Re:he's a native speaker

[zach_the_lizard]

I chose Spanish because that is the only foreign language that I have attempted to learn, not because of failing geography class. That last bit is why I don't think Europe is a country, just like some other people.

Re:It's a matter of definitions

[cbhacking]

Actually, even XP had a disk cashe (like all the other OSes you mention, it was post-caching - storing data in case you need it again soon, rather than pre-caching things that you are probably about to use). However, as I understand it, the disk cache in XP was relatively small, and XP was (at release) used extensively on machines that really didn't have the kind of RAM it wanted. Therefore, it implemented an extremely aggressive paging algorithm, constantly writing memory pages to disk but not recovering them until it absolutely had to (even if there was un-allocated RAM). Not only did this increase I/O, but it meant that switching out of one long-running application (which has been allocating and freeing memory for so long that *everything* else got paged out) takes ages.

The up-side of this is that the main running program still looks pretty fast, even on a low-memory computer. That was the goal for XP. Unfortunately, it also makes XP hopelessly bad at managing systems with the kindof RAM you see today - it will still super-aggressively swap out memory pages for inactive applications, even if it could leave stuff in main memory and just re-use a portion of the RAM. Once that RAM is freed - perhaps because the program is closed - it lacks the sense to fill it up again with cached data that you might want soon (quite possibly including chunks of the pagefile holding data for one of the background processes the user is just about to click on and would rather see activate without 4 seconds of delay to pull paged memory in again).

That said, your point stands. XP's memory management is terrible for modern systems, and aside from the difference between pre- and post-caching, all other major OSes are "memory hogs" in this manner too. Unused RAM is wasted RAM.

Re:he's a native speaker

[zach_the_lizard]

Er, I think that Europe is a country. That's why.

No simple answers

[Ancient_Hacker]

The issue is not a simple one.

On the one hand it's potentially a good idea to use "unneeded" RAM to pre-fetch possibly useful in the future disk data.

But a whole lot of apps were written without that "feature", so a lot of them already pre-fetch data. Now you have twice the amount of RAM tied up, for no benefit.

And nobody can predict the future, so the pre-fetching is speculative at best, and has no way to compensate for other tasks the user may double-click into competion with ones being pre-fetched.

Worse yet, a lot of apps look at the amount of free RAM when they decide whether to prune their working set or make other cache/purge decisions.

If all of a sudden the OS reports less free RAM, all those apps are going to make the wrong decision.

Re:It's a matter of definitions

[BikeHelmet]

XP has the same caching mechanisms as Linux. By default it uses 4MB of cache(per drive?), but if you enable the LargeSystemCache regkey, or change the setting in the System Properties, it uses all but 4MB.

The memory appears free in the XP task manager - but if you start a program up that you closed recently, it may start instantly with a minimal amount of disk IO.

If you have enough memory, I suggest turning Pagefile off. It makes it so every program is always responsive, because stuff assumed to be cached to disk is actually in memory. The downside is, programs often cache hundreds of megabytes of files they want access to, but don't currently need. This will now all be in RAM, which means you run out way easier. I've had TF2 crash from lack of memory when it hit 1.5GB/3.7GB, because of all the other programs sucking down RAM from the "VM size" column. This is the reason many sites call the XP "No Pagefile" option unstable - even though it works like a charm. When you do run out of RAM, there's nowhere for that memory to go. At least with Superfetch, you still have a Pagefile for when you do run out.

Re:It's a matter of definitions

[Anonymous Coward]

I swear... you people are fucking retards.

I doubt it; this *is* Slashdot, after all - nobody here gets laid.
 
*rimshot*
 
Thank you, I'll be here all week! Please - tip the editors!

Why do everyone complain about memory consumption?

[oycob]

These days I rarely see people putting together a system with less than 3-4 GB of memory, often as much as 8GB - and this goes for laptops as well. So if windows 7 wants to do something useful with all this memory, I say let it. My laptop runs W7 with "only" 2 GB of ram, but it runs everything I throw at it in a heartbeat. It should be mentioned that I'm not a gamer at all (except for the occasional round of minesweeper ;)), but still; your game won't utilize those 8 gigs of ram, at least not for anything really useful... Smoother user experience by caching those files accessed on a regular bacis is a big plus in my book.

Avoid "problem" by putting memory in desk drawer

[noidentity]

As someone commented on the last story a couple of days ago about this, if you don't want all your memory to be actually used, pull some of it out and put it in your desk drawer. What, you do want it all used? Well, that's what Windows 7 is doing, using all of it all the time, rather than leaving some of it unused much of the time. Oh, you only want it used for certain purposes? Why? If it's not being used for anything at the moment, using it for something is clearly better than that. And that's what Windows 7 (and Linux) do! If a more important use for it comes along, it repurposes it for that.

Re:Disk Cache?

[ClosedSource]

It might be irrelevant if we were discussing the performance of compilers on Windows vs. Linux, but that's not the topic. The topic was cache performance.

But to comment on your off-topic statements: Of course, you don't have to use VS to create CLI or GUI apps on Windows although most developers do. There are command-line versions of the compilers. Likewise, you can develop windows applications using compilers from other vendors as well.

For Linux there's a lot more than GCC available.

Re:he's a native speaker

[blind biker]

That Anonymous Coward is a very prolific poster.

Re:If only the cache were actually -good-

[mdarksbane]

That'll be great the one time I launch Chrome/firefox this week.

Pretty much the only time a web browser gets closed on my system is when flash hangs. Same for almost all of the programs that I use daily - why would I ever exit them, when I'm just going to want them again in five minutes?

Re:Noooo

[Nutria]

get this: _I_ dont NEED harddisk reads in the middle of my activities.

Then go back to DOS (or buy a quiet HDD). The rest of us understand that this is a multiprocessing OS which works even when you don't.

Idle memory is worthless.

[Low Ranked Craig]

All good operating systems do this. My Mac, for instance has "inactive memory", which is not exactly the same as Windows, but close enough. If your memory is free, it's not doing anything for you. End of story.

Re:It's a matter of definitions

[Billly Gates]

Linux uses caches and buffers

Ubuntu is in orders of magnitudes far superior than Windows Vista running superfetch on my laptop with only 2 gigs in this area.

Firefox, netbeans, and openoffice runs under 1 gig and opens within half the time compared with running their win32 counterparts in vista which maximizes all the ram and then swaps them like crazy.

I do admit I am not a kernel hacker or anything but it seems superfetch is bad in the fact that Windows just pages everything to disk when it doesn't need too, which in return slows down superfetch as it likes to fill the ram, which than swaps it even more aggressively to disk and the cycle repeats. At least this problem seemed to be there in Vista and I hope Windows 7 is not as braindead. With machines with 4 gigs the problem is not nearly as bad but a laptop with a slow hard drive with only 2 gigs its silly.

I am shocked that Firefox and Netbeans together are less than half the ram in ubuntu. After loading once they open almost instantaneously again as Linux will store these program files in its cache.

Re:It's a matter of definitions

[Billly Gates]

The problems with superfetch is when Windows begins to swap aggresively to disk as it pretends to be running out of ram when its fine. It seems to me the same programs generally feel more responsive with Linux caching than they do with Windows Vista with super fetch. Its nice not waiting on the disk so much in which super fetch is supposed to solve.

Re:Why do everyone complain about memory consumpti

[Billly Gates]

With older laptops and netbooks ram is typically between 1 - 2 gigs. Guess which OS runs better and does not swipe to the hard disk constantly?

Ars

[FartKnockerz]

While catering to the Apple fan-base a little too much for my tastes -- hits it right on.

It is scaremongering.

.

Cache is there for a reason -- idle RAM is useless RAM. Now, do I think that Windows allocation mechanism for idle RAM is right on? Of course not. It could use some tuning. It could have used some tuning since XP -- but is idle RAM stupid? Yes.

Anything that can update the pipeline between process/thread interaction with data before it has to go to disk -- is preferred.

However, Windows needs to be smart about what it caches and keeps in the pre-fetch. Frequently used and accessed rather than conditional stale.

That's the point.

Re:You're lost the point

[dbIII]

Memory leaks are the problem, especially with legacy software where the vendor is long gone. It is really annoying when a program is so badly broken that you have to feed it a command string of six numbers in octal every 108 minutes to purge memory or the bloody thing crashes.

Unused hardware is useless hardware...

[macshome]

Just like CPU utilization, idle RAM usage is a terrible performance metric. I'm constantly amazed by all the people that want to keep CPU and RAM usage to a minimum, particularly on servers. When I worked in a large enterprise I used to freak my Windows colleagues out by running my Mac OS X and Linux servers at 100%. It's not the % number of use, it's the loads that matter and learning how to size a server is more complicated than the Windows Task Manager.

In an ideal world your system would use every scrap of it's hardware to complete each task as quickly and efficiently as possible.

Re:Unused hardware is useless hardware...

[Eskarel]

100% continuously is pretty ballsy unless you really really know your server loads.

Re:Unused hardware is useless hardware...

[macshome]

My Xserves would run at 100% when they were doing a lot of postscript processing on the print queues, but on average the UNIX stuff there was loaded far more heavily than the Windows servers. (2003r2 at the time.)

The Windows guys would order more hardware when they got to 60% CPU load. This was hard for me to grasp at first when I took over the Citrix farms. Windows does actually have nice performance instrumentation and nice documentation to go with it.

The only person that loaded Windows servers harder than I did was my buddy who managed the MS-SQL OLAP cubes. When they processed a cube it would peg out 8 and 16 way Itaniums at 100% for several days.

Re:Unused hardware is useless hardware...

[Eskarel]

Oh, it can be done, you'd just have to be sure you weren't going to be getting load fluctuations.

Re:Unused hardware is useless hardware...

[Chili-71]

I run the Epic servers at 85% CPU and nearly 100% memory. However, having said that, I must admit that Epic runs on AIX 5.3 on a p570 server. I would NEVER try to run my Windows system at those levels. At 50% CPU and memory, Windoze (I have XP for stuff that needs windoze) starts to have really issues.

Not all round pegs will fit in square holes.

Re:It's a matter of definitions

[Eskarel]

I think your confused.

The old office quick starter, and the adobe acrobat version of the same thing were like that. They started applications at startup which would use up system memory(they were real applications actively running and had the same priority as any other app) to make those applications appear to start up faster. That was(and in the case of adobe at least is) a cheat and a pain in the ass. Loading an application in the background so that if I try to use it it's already running is a bad thing.

On the other this is not like this. Windows tries to guess which data from the hard drive I might want want in the immediate future. It then pulls this data into memory when I'm not trying to use the disk for something else. If it's wrong, it flushes the data away and no harm is done, but if it's right then I've saved the time taken to pull that data.

Seems to me that unless it's designed really poorly(paging loaded applications to do this, trying to cache disk when you're using disk IO for other functions, etc) then little to no disadvantage for a chance at great advantage is a pretty good thing.

Disk Trashing is bad - period!

[MindPrison]

The constant disk trashing in Vista & Windows 7 - is the ultimate Achilles heel for these OS's. You can fany it up with your wiseness about coding until you turn blue from the waiting, it still makes the machine hopelessly slow, not to mention the mental pain aquired from the constant disk-trashing-sound. Not to mention the ever-lagging-soggy response you have because the harddisk is "doing something else all the time".

Why is it, that Linux with X11 & Gnome - takes less than 27 seconds to boot on a BRAND NEW computer, and the SAME brand NEW computer...Windows 7 fights itself through disk-trashing-hell+preload-to-infinity for 5-7 minutes? I was READY for windows 7, I heard good things about it, I tried it for a WHOLE WEEK...until I almost caved in, and even though it was HELL to get all my hardware up and running with Ubuntu 9.10 (which is the shitties Ubuntu yet, hardware wise)...I STILL prefer to run Linux...compared to almost wanting to KILL my computer over the constant disk-activity!

I know I can turn it off, but Microsoft turns it ON every freaking update anyway. I'm also tired of the constant nagging - on what I should have ON / activated / turned off etc...after every update (which is nearly every day), and constant nagging about my software trying to run (because I want to run it).....MY GOD WHY CANT MICROSOFT return to Windows 2000 days....?

Re:Disk Trashing is bad - period!

[amliebsch]

Zounds, man, what are you autostarting that causes windows to take 5-7 minutes to load? My Windows 7 loads to usable desktop in about 45 seconds, and that's on a Dell Mini with an Atom CPU and 1 GB of RAM. And I haven't disabled anything.

No offense but I think you're doing it wrong.

Re:Disk Trashing is bad - period!

[MindPrison]

Non taken.

I'll forward that to Asus. ;) You guys are doing something wrong!

Re:Disk Trashing is bad - period!

[shallot]

Why is it, that Linux with X11 & Gnome - takes less than 27 seconds to boot on a BRAND NEW computer, and the SAME brand NEW computer...Windows 7 fights itself through disk-trashing-hell+preload-to-infinity for 5-7 minutes? I was READY for windows 7, I heard good things about it, I tried it for a WHOLE WEEK...until I almost caved in, and even though it was HELL to get all my hardware up and running with Ubuntu 9.10 (which is the shitties Ubuntu yet, hardware wise)...I STILL prefer to run Linux...compared to almost wanting to KILL my computer over the constant disk-activity!

There must be something wrong with your particular setup, because for example my clean Windows 7 install booted in 21 seconds to the login prompt, and 6 seconds between logging in and the disk settling down. This is with a two year old SATA disk, 2 GB DDR3 RAM and a 3 GHz quad-core processor. After installing some basic applications such as MS anti-virus, NVidia drivers, Firefox etc and some things like Steam, Cisco VPN, etc., the boot time became longer by about a third, but the OS is still just as usable. It's completely comparable to my Debian install that I continue to dual-boot on the same hardware.

Re:It's a matter of definitions

[sydneyfong]

Linux does the same thing, as far as I know, and you don't see anybody calling Linux a memory hogging OS.

Just FYI, a while ago Andrew Morton was flamed by some people for having the kernel cache too much, and a debate similar to this one ensued. Eventually he had to add a "swappiness" option to the kernel to make the complainers happy.

So yeah, Linux indeed does the same thing, but people flamed anyway.

Re:If only the cache were actually -good-

[m_pll]

Now, it would be nice if we could see what's in the cache in Windows, but you can't.

If you're really curious, you can download windbg and run !memusage from a local kernel debugging session (or using livekd). You can also use meminfo -a -f, though you'll probably need to post process the output to group used pages per file like !memusage does.

It would nice if you could peg a file to the cache, in Windows, but you can't.

Not that I would recommend that, but you can write a small app that maps a file into memory and VirtualLock's the entire view. That will make sure the file's pages remain resident.

Now you might think that after a day of builds and recompiles, at least some of that stuff would wind up in cache, but it feels like it doesn't.

It's also possible that something else makes things slow. You could profile the build process using xperf, or at least check whether these files are actually being read from disk using resmon.exe or some other disk tracing tool.

Even if it turns out that the files are in fact not cached, it's not usually the operating system's fault. Something must have pushed the cached pages out of memory. For example, maybe Visual Studio itself (or the tools it spawns, like the linker) temporarily consumes a lot of memory during the build, eating into the cache. (Though in that particular case, Superfetch should eventually reload the files back into memory)

Re:PAE?

[nhytefall]

Seriously, PAE? Come on now...

Even with PAE, then yes, you get the privilege of having your 32-bit OS running on 64-bit hardware access more than 4GB of RAM, but why? Haven't gotten a working crack of Win7?

At this point in the game, those that still run 32-bit OS's on hardware from 2 standard lifecycles ago just need to give it up and stop. 64-bit is where it is at.

Re:Disk Cache?

[ClosedSource]

Your experiment simply can't provide any insight into caching "quality" on Windows vs. Linux.

You're changing OSs and your changing applications. You can't isolate the difference in performance due to the applications and the difference in performance due to the cache.

The only thing that you can conclude was that your experience indicated that VS on Windows didn't perform as well as GCC on Linux. That's all.

Re:PAE?

[Anonymous Coward]

Except PAE is supported on all Pentium Pro and later Intel CPUs.

I don't know where you get the idiotic idea that it only works on 64-bit hardware.

Re:It's a matter of definitions

[DavidTC]

I think we need a 0 mod.

0: Redundant information that any objective observer would conclude is blatantly obviously, and normally get a Redundant mod just for stating something akin to 'A is A', yet the post was a response to a total fucking moron who apparently didn't understand that rather basic information.

But that's clearly too long to fit in the dropbox. Ah well.

'to make Windows appear faster.' Jesus H. Christ.

Windows is using my system resources to make my computer appear to be faster! Those bastards at Microsoft have struck again! How dare they make things happen faster on my computer as some sort of optical illusion of speed! (Because it's not like the speed of a computer is actually defined by 'how fast things happen'. That's crazy talk.)

You know, it's an interesting thought experiment: What if no work had ever gone into making computers 'appear' faster? If through computer history, we'd actually made them faster like normal, but put absolutely no effort into caching or optimizing or offloading processing or DMA or anything, just making processors faster and faster?

I suspect, right now, we'd be trying to figure out how to play our fancy new 'MP3s' on our eight-core-processor RAID-array setups, and hoping, one day, we'll get processors fast enough to handle postage-stamp-size video, or possibly even redraw our entire video screen at once.

Re:If only the cache were actually -good-

[shutdown -p now]

Out of curiosity, is this on XP, or Vista/7? Their caching strategies are very different...

Re:You're lost the point

[dirtyhippie]

what on earth? what program do you have to feed octal numbers to so it purges its memory?

Re:It's a matter of definitions

[amliebsch]

Except it simply doesn't do that as you can easily demonstrate with test case programs. Memory used for cache is used but not reserved, so if a program needs it, it will simply free some of the cache and give it to the program. No swapping required. Swapping programs in order to maintain the cache would be idiotic, but Windows doesn't do this.

Re:Avoid "problem" by putting memory in desk drawe

[caywen]

I agree, and it's amazing to me that all this hubbub doesn't surface around other areas where this is happening. OS's are using the GPU more both for UI and accelerating computing-intensive tasks. They use unused network bandwidth to trickle down updates and other resources from the Internet.

I suppose people will bitch if an OS uses idle time on one of 3 unused CPU cores? Or recording video of you while you're not using your webcam?

Oh wait, strike that last one.

Re:If only the cache were actually -good-

[jack2000]

Now ONLY if it had a spiffy cache configuration control menu. Why not let ME choose what it should cache and what not? Is the almighty cache dev team so enlightened so that they choose instead of me?

Re:If only the cache were actually -good-

[jack2000]

You can still use a ramdrive, you should look into "Dataram RAMDisk", it's free and fully configurable.

The problem is shitty logic.

[miffo.swe]

It seems like the problem lies in the same crappy logic handling memory in Windows 7 as in Vista. Superfetch can work but the logic has to be extremely good and Microsoft arent capable of such feats.

People notice their computer slow down to a crawl, look at memstats and see most memory used. See that little disk light illuminating bright as a star and condlude something is probably wrong in memory world. The problem isnt that most memory is used, its that its in use by a schizofrenic alzheimer despotic whirlhead.

Sorry dude - lost joke 4815162342

[dbIII]

Sorry, just trying to make the "LOST" fans laugh over the silly deadman switch thing which involved typing numbers in every 108 minutes. It made me think of some NT4 machines with crap third party software on them that needed a reboot every week or they would run out of memory and crash.

Re:If only the cache were actually -good-

[zippthorne]

Ubuntu's plan, if i read correctly, was to take advantage of periods in the boot cycle during which no IO was occurring to cache "frequently used programs" or just "stuff that's going to be needed during the boot process later" as determined by some kind of heuristic.

it shouldn't have added to the boot time because it was supposed to take place while the cpu was occupied, and presumably used direct memory access features to get the bits into the RAM.

I believe you could also add files to the list manually, but the important thing about this feature is that it does not specify what should stay in RAM, only what should be cached "if there's time" during boot.

I was hoping someone would know if there was any feature like that (other than /dev/ramN, which is inferior to tmpfs in every other way), and if there was any feature that would show what actually is cached. I'm definitely curious about it, seems like something that would be found under sysfs or procfs if it exists.

Re:Avoid "problem" by putting memory in desk drawe

[noidentity]

Well, unless I have unlimited bandwidth, I don't want my OS downloading crap or my web browser pre-loading web pages. There is a downside to doing more work than normal, so if anyone has a legitimate complaint, it's about the extra power and wear this pre-reading of hard disk content results in (and maybe even wear on the memory, though I doubt it's even measurable).

Heh, maybe we should bitch that the OS uses the entire display, even when there's no window in some portion of it. Or that it caches things on the hard disk.

I think Superfetch just needs to be more visible.

[Z80a]

Like being some sort of "trash folder" easy to acess on the control panel, where you can see exactly what files are being cached and be able to delete and put files there.

That would make it clear to the users how many ram is the superfetch using, and also allow em to add files when they actually need it instead of waiting windows to do so.

That would make the user have a better notion of "why i have low ram", because they would see the superfetch working.

Submitted stories

[furby076]

You know. I am tired of my stories getting rejected. I submit this story on Friday, it gets rejected, only to have Timothy post the same exact story on saturday. WTF. Not the first time this happened.

Re:he's a native speaker

[DaVince21]

Actually, I've seen that exact comment before. It's probably half a joke.

Re:Byte vs. megabyte

[Curate]

What's your proposed alternative algorithm for zeroing out a million bytes of RAM?