Slashdot Mirror
Windows Memory Manager To Introduce Compression
jones_supa writes: Even though the RTM version of Windows 10 is already out of the door, Microsoft will keep releasing beta builds of the operating system to Windows Insiders. The first one will be build 10525, which introduces some color personalization options, but also interesting improvements to memory management. A new concept is called a compression store, which is an in-memory collection of compressed pages. When memory pressure gets high enough, stale pages will be compressed instead of swapping them out. The compression store will live in the System process's working set. As usual, Microsoft will be receiving comments on the new features via the Feedback app.
Welcome to 2014
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Welcome to 2013! as it was then compressed memory was introduced in Mac OS X.
- Henrik
- when the Shadows descend -
This seems eerily similar to what SoftRAM was trying to do in the mid-90s. Anyone remember this? "Double Your Memory!" was it's claim and in fact the tagline on the box cover. This was back when RAM cost a fortune and everyone needed more than they had in order to run Windows 95. The company made a killing... at first.
https://en.wikipedia.org/wiki/SoftRAM
I actually worked for them, and I saw the whole thing happen from start to finish. It was quite a wild ride. Mark Russinovich and Andrew Schulman took a particular offense to the software and set about publicly dissecting it and working feverishly to prove that it didn't work. They thought the whole thing was a scam. I personally witnessed tests that indicated it was doing exactly what it said it did - however it was difficult to prove any worthwhile effect under realistic working conditions. It seemed that the primary problem was that the program needed to reserve a chunk of memory to do it's thing, then it had to make intelligent decisions about what to put in there. If it was wrong (i.e., it compressed something that the user was going to close anyway, and the user opened a new program instead of retrieving the compressed one), the memory was wasted and overall performance (of opening the new application) was diminished. The reduction in overall memory at the outset may have been putting a strain on the system which the codec was unable to outperform. To aggravate things, the software also performed a few well-documented registry tricks to optimize the pagefile settings which led critics to claim that is indeed all that it was doing.
The proof I saw, for example, if you made a spreadsheet with millions of 1s in each cell, then made a cell calculating the total of all the cells, with SoftRAM, the calculation would take a quarter of a second. Without SoftRAM, a ton of the data got swapped to disk and the calculation took like 30 seconds. However, as soon as you put realistic data into the spreadsheet, the improvement basically disappeared because it wasn't compressible enough with the algorithms they were using. They actually hired a very famous compression expert at the time, who liked to talk a lot and bill them at something like $350/hour or something crazy and it didn't seem to help at all.
Eventually the company lost a class action suit and had to refund millions back to customers. They were never able to recover, despite using their wealth to acquire and improve various products. A few of the products they put out were good, like the Mac RAM management tool (though it pre-existed, and really, the company ruined the design and marketing for it), others (like BigDisk which faked your system into believing multiple disks were one volume) had problems and could be extremely dangerous if used incorrectly.
Ahh, good times.
this is a necessary step on any hardware that doesn't have virtual memory, regardless of operating system
That doesn't have virtual memory AND uses a flat memory model (i.e.: where there's a single huge continuous address space)
If the OS needs to move memory around (paging, etc.) the only solution is to change the pointer which need to point elsewhere in memory, hence the complicated handles and pointer on Mac's Classic System, on 68k PalmOS, etc.
Meanwhile, the PC's 286 also lacked a virtual memory (that did only came later with the 386), but used (and abused) the protected mode's segmented memory as a "poor's man virtual memory".
Protected mode memory was accessed through a segment: a "handle" pointing where the chunk actually stays in memory. (A bit more complex than the real mode segment of 8088/8086 which where just spead 16bytes appart).
The soft doesn't know much, it only uses the handle it was assigned to use. If the OS needs to move memory around, it just maps the segment to a different address space. The soft doesn't know and keeps using the same handle as before.
I'm not saying that the 286 architecture was better, just explaining a bit why Intel choose to stick with segments in protected mode.
(in fact the 68k architecture was better, being 32/16 bits hybrid and being able to handle pointer mapping any position in a flat memory representation, whereas the 286 was pure 16bits and required a mumbo-jumbo of segment to handle anything bigger than 64k)
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
To be more precise:
- ZRAM create a block device that's compressed. A bit like a regular ramdisk, except that it is compressed with LZO on the flight.
It can be used for anything that a block device can be.
Traditionnaly that has been compress swap in-memory, but could be used for anything else (you could put a temporary file system on it).
Swap-on-ZRAM effectively doubles the amount of RAM: allocate 256MiB for ZRAM, get probably ~512MiB of swap on it. i.e.: you can hold extra 256MiB in-RAM.
The draw back is that swap has no concept of ZRAM and can't intelligently fallback to harddisk. You just give some swap partition on ZRAM and on HDD. All the swap are filled according to their priority.
Thus you can end-up with poorly compressible data on ZRAM, or with older data that's seldom using on ZRAM while the more used data is swapped to HDD.
- Zswap : puts an extra compression stage in the swap system between RAM and Swap. Instead of swapping out memory straigh to disk-based swap, swaped-out pages are first compressed and put in a compressed store in-RAM, then once this store is full, the least-used compressed pages are sent to disk. as the swapping system is fully aware of this (it's an actual extra layer in it) it will correctly elect to write to disk least recently used part of the compressed stage.
Another advantage is that Zswap can use any compression algorithm supported by the kernel. That includes LZ4 which is blinding fast and is usually IO-bound.
That means the CPU load doesn't suffer much, and in fact Swap-performance improves thanks to the saved bandwidth.
- Zcache : like Zswap. But instead of being an extra layer added only inside the swap-mecanism, Zcache can add similar intermediate store to other projects too (file cache, etc).
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
I have a Mac and have therefore had compressed swap for some time now. Theoretically, it's much faster than swap, even if you have an SSD. But there's a tradeoff. When swapping, the disk is busy, but the CPU is free to do other work, although things bog down a lot when thrashing happens. When doing compressed swap, the memory management hogs the CPU, which means it's not free to run other programs, and the system slows down. And thrashing still happens. It's just that your laptop heats up more when it's happening, and things don't get any less sluggish.
Of course, the biggest problem is Safari. I'll get Safari Web Content processes taking up 10GB or more. There's obviously some kind of run-away memory leak going on. Always when my system bogs down, it's Safari that's taking up too much RAM. Quit Safari, and the system becomes responsive again.