Slashdot Mirror
Windows Memory Manager To Introduce Compression
jones_supa writes: Even though the RTM version of Windows 10 is already out of the door, Microsoft will keep releasing beta builds of the operating system to Windows Insiders. The first one will be build 10525, which introduces some color personalization options, but also interesting improvements to memory management. A new concept is called a compression store, which is an in-memory collection of compressed pages. When memory pressure gets high enough, stale pages will be compressed instead of swapping them out. The compression store will live in the System process's working set. As usual, Microsoft will be receiving comments on the new features via the Feedback app.
Welcome to 2014
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Nothing slows down memory access like compression....
Do not take Edge to paypalsucks.com It can open a tab that opens a box that says to call a number because of a problem with your PC. It will not allow you to close Edge using the X in the corner or hitting Alt-F4 or going down to the taskbar. If you choose to close Edge with Task Manager, the next time Edge opens it will load the problem website again.
I tried looking in Settings for a way to uninstall and install Edge, but turned up nothing.
"As usual, Microsoft will be receiving comments on the new features via the Feedback app" ...and ignoring them.
Welcome to 2013! as it was then compressed memory was introduced in Mac OS X.
- Henrik
- when the Shadows descend -
If your Win10 device starts to melt, don't worry. That's just the CPU compressing/decompressing as fast as it can.
Gee, an Apple product did this in the 90's, compressing memory segments assigned to processes not currently executing.
(see, for example, https://www.usenix.org/legacy/...)
The same product was Apple's first to use pre-emptive multitasking,
The product? Newton.
This will be incredibly useful on embedded devices that use SSDs or Flash memory.
Of course, SSDs are growing in size all the time and you have a larger chance of your drive straight failing over running out of useful space from wear-levelling.
So the biggest advantage here will really be speed related.
The overhead of the compression is likely low enough that it MIGHT just be faster than SSDs for many small unrelated clumps of memory, but absolutely faster than HDDs. Whether it will be true of larger memory sets is another question.
Is there any actual benchmarks out there that have tested these sorts of things on Linux? I see drinkypoos link up there that shows it has been in Linux for a small time now.
I still won't be getting it.
At least not until all those dedicated hackers have prodded and poked the shit out of it and found out other spying tactics used by WiNSAdows 10 and patch them out.
I don't want my bandwidth being used without permission, even more so if I disable it and it stays on! Believe it or not, a large number of ISPs place restrictions on upload bandwidth, and even more so overall bandwidth used per period. (monthly usually)
When I turn things off, I expect them to stay off permanently until I say otherwise.
I think many offices would also agree. (they say the enterprise version will allow full disabling of services, but I don't trust them at all)
Awesome. Windows 10 is like Linux 2012!!!
That's disk compression not memory compression.
Reverse engineering existing open source libraries and rewriting so the original source is unrecognizable seems to be a lot of work.
Um, DriveSpace was a disk compression method. It was like treating a partition like a zip file.
This has nothing to do with compressing data in RAM, which the artcle is about
"When life gives you lemons, don't make lemonade. Make life take the lemons back!" -- Cave Johnson
Doesn't this make ECC memory even more needed ?
Since compression is the process of reducing redundant information, any bit flip could kill the entire compressed unit.
That's disk compression not memory compression.
What is it when a Solid State Drive (SSD) is used?
That was a disk compression scheme, while this is virtual memory compression, more like RAMDoubler from Connectix which worked quite well (on Macs, anyway). It's possible Microsoft acquired some of that IP when they bought VirtualPC.
Blank until
, which introduces some color personalization options, ...
You no longer have to put up with the blue screen of death. Now you have the option to have speckled, sparkled, opalescent, translucent, scintillating, coruscant, flourescent, effervescent blue screens of death.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
I think you mean things like MemoryDoubler and stuff that was around at that time.
DriveSpace was purely disk-based. There were products that compressed pages in-RAM, and have been since the DOS days, and still are, and are present on every OS if you look hard enough.
Still disk compression :D.
This seems eerily similar to what SoftRAM was trying to do in the mid-90s. Anyone remember this? "Double Your Memory!" was it's claim and in fact the tagline on the box cover. This was back when RAM cost a fortune and everyone needed more than they had in order to run Windows 95. The company made a killing... at first.
https://en.wikipedia.org/wiki/SoftRAM
I actually worked for them, and I saw the whole thing happen from start to finish. It was quite a wild ride. Mark Russinovich and Andrew Schulman took a particular offense to the software and set about publicly dissecting it and working feverishly to prove that it didn't work. They thought the whole thing was a scam. I personally witnessed tests that indicated it was doing exactly what it said it did - however it was difficult to prove any worthwhile effect under realistic working conditions. It seemed that the primary problem was that the program needed to reserve a chunk of memory to do it's thing, then it had to make intelligent decisions about what to put in there. If it was wrong (i.e., it compressed something that the user was going to close anyway, and the user opened a new program instead of retrieving the compressed one), the memory was wasted and overall performance (of opening the new application) was diminished. The reduction in overall memory at the outset may have been putting a strain on the system which the codec was unable to outperform. To aggravate things, the software also performed a few well-documented registry tricks to optimize the pagefile settings which led critics to claim that is indeed all that it was doing.
The proof I saw, for example, if you made a spreadsheet with millions of 1s in each cell, then made a cell calculating the total of all the cells, with SoftRAM, the calculation would take a quarter of a second. Without SoftRAM, a ton of the data got swapped to disk and the calculation took like 30 seconds. However, as soon as you put realistic data into the spreadsheet, the improvement basically disappeared because it wasn't compressible enough with the algorithms they were using. They actually hired a very famous compression expert at the time, who liked to talk a lot and bill them at something like $350/hour or something crazy and it didn't seem to help at all.
Eventually the company lost a class action suit and had to refund millions back to customers. They were never able to recover, despite using their wealth to acquire and improve various products. A few of the products they put out were good, like the Mac RAM management tool (though it pre-existed, and really, the company ruined the design and marketing for it), others (like BigDisk which faked your system into believing multiple disks were one volume) had problems and could be extremely dangerous if used incorrectly.
Ahh, good times.
My concern with any memory management strategy under Windows is that even the current, disk-based virtual memory system is horrible at determining the "memory pressure" statistic. Under Windows 7, when I have a memory-intensive operation running, I'll hear the disk grinding away paging the whole time, while the system monitor shows physical memory usage at 60%. Even if the other 40% is disk cache, I'm pretty sure the foreground process should take precedence.
The other frustrating scenario is in sleep mode: after an overnight sleep,you can watch the physical memory line go from near zero back to where it was before the sleep as the disk grinds away paging things back in. That's hibernation, not sleep! My suspicion there is a feature which gets the machine hibernated while sleeping, to recover in the case of a power outage. The feature pretty much kills the usefulness of sleep, though, if every wake is a wake from hibernate.
Long story short, I'm pretty sure that this new compression feature means that Windows will simply keep itself to an even tinier corner of the physical RAM, while wasting CPU cycles in addition to disk accesses.
"Because Science" is one step from "Because old book". Try "Because of my experiment testing my falsifiable assertion".
A solid state drive (SSD) has no disk, but a collection of memory chips. So whatever compression it use must be memory compression. It can't be disk compression.
As usual, Microsoft will be receiving comments on the new features via the Feedback app.
After our offices relocated we started having a strange unexplained auto reboot of windows 7 systems. Seemingly random, different machines on different days, whether overnight jobs were running or not it did not matter. But every other day one machine would have rebooted overnight. Took enormous amounts of digging, but the clue was that it was always between 12 midnight and 12:30 AM. Finally localized it to some service called "Windows Experience". Apparently it was introduced when Vista came along to pop up the dialog "this application has crashed. Do you want to relaunch it compatibility mode?". This would collect all the crashed programs of the day and phone home at 12 midnight. For some reason it did not cause any trouble in the old location. But for some reason in the new building the phoning home app would crash the system and trigger an auto reboot. Even our IT could not figure it out. It took some really determined (or pissed off ) developer who kept digging through the logs to locate the offending service.
So before Microsoft introduces another feedback app, it should first let us all know how to disable it.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
The first one will be build 10525, which introduces some color personalization options
Will I finally be able to have active/inactive windows coloured differently enough that I can tell which is which at a glance? That's been missing since Vista (unless you're willing to disable Aero)
systemd is Roko's Basilisk.
What the F U C K are you talking about? Seriously, get the fuck out!
A Solid-state Drive (SSD) (also known as a solid-state disk)...
I get what you're going for but memory and disk both imply specific functions rather than just physical properties. A solid state disk is a device intended for the use of storing data rather than performing the functions of facilitating working program data. Contrary to your assertion it must not be memory compression as the function of the device is that of storing data at rest rather than in motion as memory does.
Storage compression.
You must not like brain teasers.
From reading the comments I've learned that:
- Only Micro$$$haft's OS is bloated and inefficient and bad enough to need this
- Linux and OSX have had this for years, Mrw00wla$chlong is so late to the party lololol
The compression store will live in the System process's working set.
Exploit in 3..2..1..
All kidding aside, I hope Microsoft (and Apple and other vendors before now) have given this more thought than their adversaries.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Secondary storage device compression. Now are you happy? It's nonvolatile, write-cycle-limited, rather slow memory chips being used as a secondary storage device, in the same manner a disk is used.
I thought Windows 10 handles memory great from install to running it. RAM seems to be in check and it has a small install foot print then Win 8. But I can say this, the Edge browser tops even Chrome with RAM consumption in all three PC's I have Win 10 installed. My HP Stream 11 with only 2 GB RAM suffers the most with this. I finally abandoned Edge for IE 11 on that device. IE 11 is really the only browser that does not eat up the RAM as bad. Chrome is still a hog when you start opening tabs up and Firefox is OK unless you install Flash player. Yea, speaking of Flash, its really part of the problem and It does help Edge do better if you go into settings and turn off Flash.
because the Feedback app no longer works on most W10 systems, and they have yet to fix it. (I wasn't able to send any feedback about it.)
Like many others, I got horrible flashbacks to the tools like RAM Doubler of the nineties when reading the headline. I remember seeing many Windows machines having their performance absolutely killed by these sort of tools. However, the approach of using it as a sort of storage tier in - between normal primary memory and disk swap space actually could make great sense. Of course uncompressed memory will work much faster, but if well - implemented, aces sing compressed data in memory could be significantly faster than aces sing the same data from disk. My normal advice would still be, that RAM today is so cheap, that you should always have enough to avoid paging, and that would also apply to teqniques like this, but for the many cheapskate users that save a few quit on RAM in their PC, this could probably be a good thing.
A disk compression technology from DOS 6+, Stacker was also a drive compression product, i don't believe either supported keeping the windows swap file on a compressed volume, at least initially, so they seem DoubleIrrelevant to the discussion.
The article doesn't explain which compression algorithm is being used. It may very well be the zip compression format. .NET has a in-memory compressor that uses the zip compressor.
http://www.codeproject.com/Articles/14204/Better-Than-Zip-Algorithm-For-Compressing-In-Memor
.. call me a little suspicious...
Does this mean I have to put HIMEM.SYS and EMM386.EXE back into my config.sys file? I think I still remember some of the MS-DOS edit commands.
You are welcome on my lawn.
No you are just stupid, that's all.
SSDs are not made of memory chips, they are made of flash chips. Nowhere near as fast as RAM.
When you've been using Linux since 1996 like me
What makes people like you think that the O/S should be held responsible for buggy applications? When I read that sort of emotional nonsense about operating systems it just makes me think the author hasn't got a clue about anything outside the intellectual cage he has built for himself.
Inertia is a powerful force. It's not always easy to learn something new
The problem with most of the warriors in the O/S wars is they divorced windows 20yrs ago but are still bitter about the split. If you had been paying attention you would realise that windows is not the same O/S you split up with in the 90's, it grew up when XP was released, you would be wise to follow its example, nothing says "bitter" as clearly as someone trying to belittle the people who learn to like/love their ex.
Disclaimer: degree qualified software developer for 25yrs, comfortable with a wide variety of unix and windows flavours.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
But you probably couldn't put it into large-scale production until the RAM Doubler patents ran out.
On a laptop or tablet, the backlight probably draws far more juice than the CPU. So if the CPU can complete a task more quickly by not hitting the HDD or eMMC as often, the backlight won't need to be on as long, which saves power. I wonder whether this is a simple enough task to be put on the little cores in ARM's big-little configuration.
Single level storage - a single address space for everything, and let the dedicated I/O controller sort out what needs to be in memory at any one time.
In a single-level storage model, the main RAM acts as a cache for mmap'd disks. The compressed part of RAM would then act as an additional cache level, which reduces the number of capacity misses that need to reach the disk.
Or, just put 8+GB of RAM in your machine
That's fine on a recent desktop, not so fine on an older desktop with few slots or on a compact or detachable laptop with soldered-in RAM.
What makes people like you think that the O/S should be held responsible for buggy applications?
If applications are included with the default install of an operating system distribution, such as Edge with Windows 10, then of course the distributor is responsible for them. And if the bug is in the standard library provided with a compiler, it's the fault of the compiler publisher, which is often also the operating system publisher.
ObZram: Once RAM compression becomes commonplace, a memory allocator that zeroes out recently freed memory will be more space-efficient. A standard library that does not do this when under memory pressure will be considered "buggy", even if it's published by the operating system publisher.
I'm not afraid. DriveSpace had the DoubleGuard feature, which patched MS-DOS to add canaries around critical file system data structures in RAM. This saved my bacon a few times when I was developing graphics code and accidentally introduced undefined behavior.
My question is, with mid-level machines coming with 16gig of RAM, why would I need compression at all?
Because not all machines are mid-level. With a lot of smaller machines, especially phones, tablets, and detachable laptops, the 1-2 GB that comes soldered on when you buy it is all you get.
this is a necessary step on any hardware that doesn't have virtual memory, regardless of operating system
That doesn't have virtual memory AND uses a flat memory model (i.e.: where there's a single huge continuous address space)
If the OS needs to move memory around (paging, etc.) the only solution is to change the pointer which need to point elsewhere in memory, hence the complicated handles and pointer on Mac's Classic System, on 68k PalmOS, etc.
Meanwhile, the PC's 286 also lacked a virtual memory (that did only came later with the 386), but used (and abused) the protected mode's segmented memory as a "poor's man virtual memory".
Protected mode memory was accessed through a segment: a "handle" pointing where the chunk actually stays in memory. (A bit more complex than the real mode segment of 8088/8086 which where just spead 16bytes appart).
The soft doesn't know much, it only uses the handle it was assigned to use. If the OS needs to move memory around, it just maps the segment to a different address space. The soft doesn't know and keeps using the same handle as before.
I'm not saying that the 286 architecture was better, just explaining a bit why Intel choose to stick with segments in protected mode.
(in fact the 68k architecture was better, being 32/16 bits hybrid and being able to handle pointer mapping any position in a flat memory representation, whereas the 286 was pure 16bits and required a mumbo-jumbo of segment to handle anything bigger than 64k)
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
Flash is a type of memory, so you're both right.
To be more precise:
- ZRAM create a block device that's compressed. A bit like a regular ramdisk, except that it is compressed with LZO on the flight.
It can be used for anything that a block device can be.
Traditionnaly that has been compress swap in-memory, but could be used for anything else (you could put a temporary file system on it).
Swap-on-ZRAM effectively doubles the amount of RAM: allocate 256MiB for ZRAM, get probably ~512MiB of swap on it. i.e.: you can hold extra 256MiB in-RAM.
The draw back is that swap has no concept of ZRAM and can't intelligently fallback to harddisk. You just give some swap partition on ZRAM and on HDD. All the swap are filled according to their priority.
Thus you can end-up with poorly compressible data on ZRAM, or with older data that's seldom using on ZRAM while the more used data is swapped to HDD.
- Zswap : puts an extra compression stage in the swap system between RAM and Swap. Instead of swapping out memory straigh to disk-based swap, swaped-out pages are first compressed and put in a compressed store in-RAM, then once this store is full, the least-used compressed pages are sent to disk. as the swapping system is fully aware of this (it's an actual extra layer in it) it will correctly elect to write to disk least recently used part of the compressed stage.
Another advantage is that Zswap can use any compression algorithm supported by the kernel. That includes LZ4 which is blinding fast and is usually IO-bound.
That means the CPU load doesn't suffer much, and in fact Swap-performance improves thanks to the saved bandwidth.
- Zcache : like Zswap. But instead of being an extra layer added only inside the swap-mecanism, Zcache can add similar intermediate store to other projects too (file cache, etc).
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
He didn't say distributor, he said the OS, as in the files themselves. Microsoft is the distributor. A set of files is the distribution, separate from the files that make up the apps that are distributed with it.
What will this do to mitigate buffer overflows, stack exploits and other memory management bugs.
In addition to the products already listed, there was QEMM's MagnaRam - which ran on Windows 3.1 (there was also a Win95 version too).
Nice to read this news in the proper context.
I have a Mac and have therefore had compressed swap for some time now. Theoretically, it's much faster than swap, even if you have an SSD. But there's a tradeoff. When swapping, the disk is busy, but the CPU is free to do other work, although things bog down a lot when thrashing happens. When doing compressed swap, the memory management hogs the CPU, which means it's not free to run other programs, and the system slows down. And thrashing still happens. It's just that your laptop heats up more when it's happening, and things don't get any less sluggish.
Of course, the biggest problem is Safari. I'll get Safari Web Content processes taking up 10GB or more. There's obviously some kind of run-away memory leak going on. Always when my system bogs down, it's Safari that's taking up too much RAM. Quit Safari, and the system becomes responsive again.
It's called "flash memory", bub.
Nah, RAM Doubler was basically swap but not as good. It brought less rubbish memory management to early MacOS.
The page tables used in nearly all current OS implementations are designed to support the amount RAM typically found in a system 15 years ago, and waste quite a lot of cache space because of this on contemporary systems. If MS is going to update their design then I applaud them but they need to solve the fundamental issue of virtual memory page sizes being too small for any processor that doesn't have an enormous cache, lest they risk having another Windows RT debacle on their hands. Using page faults as a trigger to decompress memory pages is a rather ingenious design though, I must admit.
There are two big reasons for the "Hell no". First, in the 70s and 80s Computers were all about money. If you could afford it, you did it. If you did not have enough memory, you either re-wrote or paid for more. I'm sure we could have reserved some core to hold the LZ libraries, but that would consume more space than any piece of code we ran. Space conservation would be the second reason.
I'd say the same for the first *Nix systems as well. If you had to worry about compressing in memory you were doing something wrong.
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
Oh, please. Simple compression/decompression code fits into 1kb with another kb or so for dictionary and decompression buffers. So with a typical 50% compression rate the breakeven point is around 8kb. In the middle of 80-s computers had around 1Mb of RAM.
Great Scott Marty I have a brand new idea. let's compress the memory... How revolutionary.
Not safari. More than likely one of the extensions you have installed. Disable the extensions and the memory leak will go away.
Wasn't the Mac limited to 128Kb because that was all anyone would ever need? For that matter, most people were running DOS on PCs and that had the 640kb limit.
Wasn't the Mac limited to 128Kb because that was all anyone would ever need?
Lisa ("Mac XL", prototype mac essentially) 512k-1MB, 2x HDD expansion only.
Original mac, 128k expandable to 512k by soldering, no expansion bus. Followed by 512k version with no other changes.
All macs thereafter: 1+MB, expansion bus
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
... now make 32-bit versions to save more memory, like Linux.
For a beta-version, it would need to be at last "feature complete".
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
It's unlikely to be the extensions. When I report this bug to Apple, they too are satisfied (as far as I can tell) that it's not the extensions. I have the following enabled:
ClickToFlash (this should actually be GOOD for the memory footprint)
Sessions (because Safari natively sucks at saving and restoring its state)
Google Scholar Button (which was only recently installed and has not affected anything)
Performance of my system went to hell when Apple added this to OS X. Apparently I was already on the cusp of needing more RAM, and this pushed me over the edge. The fundamental problem with this is that the compressed pages take up space in RAM when your system is already low on RAM. Duh. I think it's more about reducing writes to SSD than it is about improving performance.
I don't need it because I just read on Slashdot yesterday that 4GB of RAM is just plenty.
Wasn't the Mac limited to 128Kb because that was all anyone would ever need?
Do you have any other ignorant statements to add?
If you're running a desktop or server on an even vaguely modern physical machine, you've got more cores than you know what to do with, and if the OS wanted to get fancy it could probably use the GPU for decompression or compression. (Also, at least with LZW, decompression is much faster than compression, and compression is something you do with memory you weren't currently using anyway, so it's ok if it's a bit slower.)
On virtual machines, it's a different game, because you're sharing the CPU with other VMs and often only have one core.. You may still want to use it, because memory may still be more scarce than CPU, and while you may have a faster disk array, it's often farther away than the disk on a physical machine.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Of course, Johnny Mnemonic was set in 2021, but still, whoa! Welcome to the Free City of Newark!
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
stolen from Linux zram
We started experimenting with the compressed memory module (zram) in Chrome OS in 2012, and deployed it fully in 2013. It's a simple approach: the kernel uses a compressed RAM disk as a swap device. It works pretty well. On average we store about 3 compressed pages into one page.
He's right, sort of.
Jobs insisted that the first Mac not have any expandability including additional RAM because he didn't want customers "mucking with the innards."
He was pretty much okay with selling them a new computer if they needed more RAM!
You don't get to leave early just because your CPU compiled your code faster or that web page loaded a bit faster.
You do if you're paid on contract, not hourly.