Slashdot Mirror

← Back to Stories (view on slashdot.org)

IBM Promises More Memory In The Same Space

Posted by ryuzaki0 on from the ibm-please-license-to-micron:)-- dept.

dcallaghan was among the many readers to write with news of IBM's announcement of new memory technology. The upshot seems to be on-the-fly compression in hardware, taking the tack of RamDoubler and other software compression utilities, but moving the actual data sqashing into dedicated (fast) chips. I hope this leaks out of "server only" land soon; I'd love to have 256MB for the price of 128 -- this would be especially nice with pricey notebook memory.

50 of 173 comments (clear)

  1. It won't last by tedtimmons · · Score: 2

    Ask yourself: how many people do you know that double their disk drives? Probably not very many, if any at all.

    A few years ago that was all the rage. Now drives are cheap and large. I expect the same to happen with memory- with all the new technologies coming down the pipeline, would you really want to hassle with a "ram doubler"?

    Even if it's in memory, you KNOW it's going to cause a bug in some program somewhere :-)

    1. Re:It won't last by whistler-z · · Score: 3

      Now drives are cheap and large. I expect the same to happen with memory . . .

      Actually, based on the recent Rambus dealings, it's very likely that RAM prices will go up, not down. Assuming this technology can do what they claim (conceivable), and not have an impact on performance (highly doubtful), this could seriously lessen the impact of Rambus's patent squablings on the end user's wallet.

    2. Re:It won't last by Daniel · · Score: 2

      As a matter of fact, I am running e2compr, the transparent compression module for ext2 (the Linux filesystem). I have been for over a year. It cheerfully handles compression with far more fine-grained control than most Windows programs (eg, you can choose individually whether to compress a file or not) and performs, so far as I can tell, flawlessly. That is, it has never corrupted data or caused bugs in other programs, and has never crashed the system even under heavy load. There was one crashing problem (that I never triggered) which was recently discovered and fixed; I don't think there are any other known outstanding issues.

      So yes, transparent compression can work just fine. The case of running out of space is a problem, but no program should blindly assume that the amount of disk space remaining is reliable unless it's running on a single-process system, and even then it's shaky. What if, while the program that needs X disk space is running, I download and unpack a file? Worse, what if another user does it on a multi-user system? Programs that break in this way are broken already, period.

      Daniel

      --
      Hurry up and jump on the individualist bandwagon!
    3. Re:It won't last by orpheus · · Score: 4

      Five major points about system memory compression:

      1. Why do you want to do it
      Is it because RAM is expensive? Okay, but RAM prices would have to climb much higher to make it worth the new boards, new architectures, and intrinsic problems

      Is it because your system will 'run faster with more RAM'? Don't count on it. Trading RAM latency for apparent RAM size will mean that a given apparent RAM size would run slower than when uncompressed (i.e. 64->128MB compressed is slower than 128MB uncompressed), and the performance gain would be variable for a given *true* RAM size (= larger apparent size), and may disappear in certain settings (Is 64MB->128 MB faster than 64MB compressed? Depends.)

      Remember, CPUs are data hungry critters, and feeding them at one end (and emptying them at the other) is already one of the biggest challenges of modern system design.

      2. Transparency is not enough. We need ultra-transparency
      Remember: any general purpose compression yields variable results with different data (and changing a bit will change the 'actual size' of a block, and hence the physical location oif the bytes within the block. Compression confounds the 1:1 correspondence between the physical and logical memory address, and the relationships between different memory addresses, so we'll need to de/compress entire blocks and cache them (more on this later)

      Without ultra-transparency, optimizing low level code becomes fraught with emergent effects. the most important thing about RAM is the capability for RANDOM access a lot of people have forgotten there was any other kind -- serial memory, bubble memory, etc. -- bucket-brigaded bits demanded very different algorithms for efficiency!

      Think of the pitfalls of straight CPU/mobo caching designs. 'Cache thrashing' can bring some fast algorithms to a molasses crawl, precisely because caching disrupts the relationship between between contiguous bytes: A slower algorthm that reuses bytes in cache is preferred over a 'mathematically faster' one that relies on massive sequential reads. Compression thrashing can do the same, and will have (multiple) cache problems on top of this..

      3. Where did I put that block?
      There is no assurance that you will be able to return a block to the same DRAM location you got it -- change a few bytes, and it may be larger (in physical length) than it was, despite having the same virtual length. This implies RAM fragmentation -- and all the associated housekeeping. And where are you going to store the record keeping? In *another* local cache? In RAM?

      You'll need lots of hardware housekeeping here. It's do-able, but without a level of sophistication that approaches predictive brancing and pipelining, you can count on extensive unexpected 'emergent effects' -- code that's slow for non-obvious reason, or bugs.

      4. Strangely, on-chip cache may be the best place to use hardware RAM compression!
      a) Hardware compression would be a small addition to the CPU circuitry, and can be run at the full multiplier speed.
      b) No need for separate chips or mobo redesign
      c) on-chip cache costs hundreds of dollars a meg (price a 512K PIII vs a 2MB PIII Xeon of the same clock speed) so extraordinary measures can be taken. It will probably improve chip yield, too.
      d) integrating compression with the prediction/pipelining/cache management/etc of the CPU, can make it more transparent
      e) L1/L2 is where you will get *huge* payoff, by 'keeping baby fed'.
      f) there's much less latency (chipset, PC trace, L1, L2, L3 cache) on-chip vs. off chip, so Adding an off-chip layer adds more latency than adding an on-chip layer

      5. The fundamental performance rule is: trade excess performance in one place to improve inferior performance elsewhere
      RAM is no longer a source of 'excess performance', it's a pressure point. Every extra 10MBps in RAM throughput shows up in the benchmarks (unlike, say HDD busses like ATA33/66/100, where doubling or tripling speed does little for system performance.

      Where's the excess speed in modern systems? It's inside the CPU -- which runs at a multiplied clock speed, has vast optimization, and is always starved for data. it's also the place where adding RAM does the most good.

      However (!) using compressed on-chip cache will require an intensive study and redesign of cache theory, unless this possibility has already been explored in conjunction with the development of the current CPU features like predictive/pipelining/VLIW. 2-,4-, 8-way associative simply won't hack it when the cache doesn't have 1:1 virtual/physical data correspondence!

      --

      If you can go to bed, knowing you did a valuable thing today, you're very lucky. If you can't... it's not bedtime

  2. Does that mean software compression is now moot? by efuseekay · · Score: 2

    So the compressing is done on hardware, cool.

    But since once compressed, we can't compress
    them any further, does this mean that any
    software compression to increase mem space is
    no longer usable?

    I like extra MB, but let's hope that they
    actually increase the physical storage space instead of just increasing "logical" storage.

    --
    Mode (3) smart-aleck mode. Press * to return to main menu.
  3. This is good for bus speeds by Sludge · · Score: 3

    This would rock on the PC, where our bus speeds are slowly reaching 133mhz. If you could send the data compressed across the bus, that is.

    Avoiding the low PC bus speeds are what 3D cards do best. You only upload the textures, and then you just send the vertices of the polygons across every time. Hell, some newer cards are even doing the calculations on some of the vertices once they've made the jump across the bus.

    This also holds true for DVD decoders.

    I wonder how viable hardware decompression is? Would it be a catch all solution for (low end) replacements for all these avert-the-PC-bus hardware cards? Admittedly, I'm not in touch with any relevant benchmarks to this sort of stuff these days.

  4. Ethics behind this by MostlyHarmless · · Score: 2

    This is kind of a tangent, but would there be any issues behind this if they were using, say, LZW compression? Yes, I now they're not. But if they did, would it be kosher? Would we be allowed to use it while still feeling ethically clean?

    This is similar to an issue that popped up when RMS spoke in Cincinnati. Someone asked if it was ok to use Transmeta's chips if the conversion layer was proprietary. The answer was that he had not heard anything about it (it was a few days after they made their big announcement) but he guessed (I think) that it might be ok, since hardware costs money to distribute (while software does not). My memory may be slightly flawed on this; don't quote me on it.

    As free software gains, we will start encountering this questions more often, with the original, simple principle of sharing software moving into a more general ethical realm dealing with intellectual property in general.

    --
    Friends don't let friends misuse the subjunctive.
    1. Re:Ethics behind this by MostlyHarmless · · Score: 2

      Of course IBM would be able to develop this without being sued by someone else. That was not the issue in question for me.

      The real problem is how the free software movement should react to things ilke this in general. Do the same ethics apply in the hardware areas as well as in the software? Obviously, we can't abstain from using any software that's patented, but we can still fight against ridiculous ones such as the one-click.

      --
      Friends don't let friends misuse the subjunctive.
    2. Re:Ethics behind this by Sloppy · · Score: 2

      Do the same ethics apply in the hardware areas as well as in the software?

      Yes and no. Yes, the same ethics apply, but no, the ethical objection against the LZW patent does not apply. Consider the enormous capital that a company would have to expend in order to manufacture these chips (the fact that it's IBM should give you a hint as to the scale), as well as the months (years) of planning and design that go into them. Does a patent search (and negotiations) seem like an overwhelming part of the development budget? (Compare this to a programmer who can inadvertantly infringe upon a dozen software patents per hour of his work.)

      Furthermore, this is allegedly transparent, where systems that it interfaces with are not required to also license the patent. Compare this to a protocol or interchange file format (e.g. GIF). If you use GIF, you have to license the LZW patent. If you use these chips, you won't.

      I don't see the patent doing any damage here.

      Same ethics always apply, but different conclusions depending on circumstances.


      ---
      --
      As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
  5. ALERT! danger! by Signal+11 · · Score: 5
    Whoah, wait, backup, slow down.. halt.

    Compression CANNOT guarantee anything better than 1:1 ratio - it is ENTIRELY dependent on the data.

    For data compression in memory to succeed, you MUST have an option to cache the "extra" memory to a swapfile incase the prediction logic fails and you run out of physical ram. If you do not, you will tank your system, bigtime.

    Sorry, but I'm very leery of any "memory compression" - it requires OS support to function. Period. You aren't going to just plug in a miracle DIMM and make it work. I hope IBM is opening the spec (it looks like they are) and that OS development people quickly embrace this, or their hardware will take a nosedive in the market.

    1. Re:ALERT! danger! by Some+Strange+Guy · · Score: 2
      Compression CANNOT guarantee anything better than 1:1 ratio - it is ENTIRELY dependent on the data.

      Precisely what I was going to say. But then I started thinking about it some more. You can't guarantee even a 1:1 ratio; purely random data will compress to slightly larger than original size, as you have to tag it.

      Stepping back for a second, though, is this REALLY a problem for most systems? I'd be willing to bet that 99.999% of the time, multi-user systems run memory images that are at least moderately compressable. The curve of pessimistic memory availability vs. actual usage would yield something akin to a MTBF, which would be a good enough guarantee for those server spaces that are looking to save a few bucks.

      Of course, memory usage limits would need to be in place to prevent some really obvious machine-crash attacks from users, but isn't that true already?

    2. Re:ALERT! danger! by QBasic_Dude · · Score: 3
      Compression CANNOT guarantee anything better than 1:1 ratio - it is ENTIRELY dependent on the data.

      This is true with random data, but most data is not random. A quote from the comp.compression FAQ:

      - The US patent office no longer grants patents on perpetual motion machines,
      but has recently granted a patent on a mathematically impossible process
      (compression of truly random data): 5,533,051 "Method for Data Compression".
      See item 9.5 of this FAQ for details.

      As can be seen from the above list, some of the most popular compression
      programs (compress, pkzip, zoo, lha, arj) are now covered by patents.
      (This says nothing about the validity of these patents.)

      Here are some references on data compression patents. Some of them are
      taken from the list ftp://prep.ai.mit.edu/pub/lpf/patent-list.

      ....
      9.2 The counting argument

      [This section should probably be called "The counting theorem" because some
      people think that "argument" implies that it is only an hypothesis, not a
      proven mathematical fact. The "counting argument" is actually the proof of the
      theorem.]

      The WEB compressor (see details in section 9.3 below) was claimed to compress
      without loss *all* files of greater than 64KB in size to about 1/16th their
      original length. A very simple counting argument shows that this is impossible,
      regardless of the compression method. It is even impossible to guarantee
      lossless compression of all files by at least one bit. (Many other proofs have
      been posted on comp.compression, please do not post yet another one.)

      Theorem:
      No program can compress without loss *all* files of size >= N bits, for
      any given integer N >= 0.

      Proof:
      Assume that the program can compress without loss all files of size >= N
      bits. Compress with this program all the 2^N files which have exactly N
      bits. All compressed files have at most N-1 bits, so there are at most
      (2^N)-1 different compressed files [2^(N-1) files of size N-1, 2^(N-2) of
      size N-2, and so on, down to 1 file of size 0]. So at least two different
      input files must compress to the same output file. Hence the compression
      program cannot be lossless.



      For data compression in memory to succeed, you MUST have an option to cache the "extra" memory to a swapfile incase the prediction logic fails and you run out of physical ram. If you do not, you will tank your system, bigtime.

      This is not true. Auxilary memory will most likely be stored on the chip itself. Data compression does not predict logic. A stream is compressed by examining it's redundancy and storing pointers back to the original match (as LZSS does), or encoding each symbol in a less number of bits (as in Huffman).


      Sorry, but I'm very leery of any "memory compression" - it requires OS support to function. Period. You aren't going to just plug in a miracle DIMM and make it work. I hope IBM is opening the spec (it looks like they are) and that OS development people quickly embrace this, or their hardware will take a nosedive in the market.

      This is not true. There are a number of hardware data compressors. MPEG is decoded in hardware by the N64 hardware, for instance. "Miracle DIMMs" are known as hardware compression units.

    3. Re:ALERT! danger! by um...+Lucas · · Score: 2

      Compression CANNOT guarantee anything better than 1:1 ratio - it is ENTIRELY dependent on
      the data.

      Nothing in life is guaranteed, you're right. But most of the plain old ASCII text files on my hard drive compress 75-90%, depending on length.

      For data compression in memory to succeed, you MUST have an option to cache the "extra" memory to a swapfile incase the prediction logic fails and you run out of physical ram. If you do not, you will tank your system, bigtime.

      Well, it's not like IBM's mandating the removal of virtual memory or swap files in order for this to work. That's what it's there fore, to catch whatever won't fit in RAM.

      Sorry, but I'm very leery of any "memory compression" - it requires OS support to function. Period. You aren't going to just plug in a miracle DIMM and make it work. I hope IBM is opening the spec (it looks like they are) and that OS development people quickly embrace this, or their hardware will take a nosedive in the market.

      Not so. Just look at connectix' RAM doubler on the Mac. It was a 300 to 400k extension that DID effectively double a systems memory, with a 3% to 5% slowdown, and NO extra help from apple. If the compressions now done in hardware, I'd expect 0% slowdown, and still no requirement that each OS explicitly support it. So long as it's built in at the chipset level, as long as the OS can communicate with the memory controller, everything should work fine, right?

    4. Re:ALERT! danger! by Azog · · Score: 3

      Memory compression (and disk compression too) are useless to me, with one exception: texture compression on video cards.

      Think about this. I have 256 MB of ram and 50 GB of disk space on my main machine. Why not, it's cheap! What I want is lower latency storage. Right now, that is expensive or unavailable, but would make a difference to the performance-sensitive apps I run.

      Furthermore, the large files that I would like to compress are already compressed: video (MPEG), sound (MP3), and image files (JPG). On my 40GB server, I have 17 GB of MP3s, 1 GB of other files, and the rest is empty space! Why bother compressing that 1 GB just to get half a gig back?

      From what I hear, even people who do non-linear video editing are editing compressed MPEG directly these days, rather than using uncompressed AVI or other formats.

      So... What applications could actually benefit from this?

      Maybe web servers with thousands of text files in RAM? Maybe people working with extremely high resolution bitmaps or uncompressed video?

      Torrey Hoffman (Azog)

      --
      Torrey Hoffman (Azog)
      "HTML needs a rant tag" - Alan Cox
  6. How fast can it be? by exploder · · Score: 4


    IANAHardware Engineer, but it seems to me that RAM is already designed to do one simple thing (okay, two things: peek and poke) and to do it absolutely as fast as possible. This technology inevitably will degrade RAM performance by a finite amount. Is their chip fast enough that this degredation will be negligible? If so, then this will be Extremely Cool. If not, then no thanks, I'll just shell out for the extra RAM. Of course, the economics on a huge server with 100GB of RAM are most likely compeletely different.

    --
    Yo dawg, I heard you like the Ackermann function, so OH GOD OH GOD OH GOD
    1. Re:How fast can it be? by Christopher+Thomas · · Score: 2

      IANAHardware Engineer, but it seems to me that RAM is already designed to do one simple thing (okay, two things: peek and poke) and to do it absolutely as fast as possible. This technology inevitably will degrade RAM performance by a finite amount. Is their chip fast enough that this degredation will be negligible?

      The saving grace for RAM compression is that DRAM is very slow by logic standards. As long as they're doing something simple like short-run run length compression, the compression/decompression could easily be fast enough to not be noticed compared to the DRAM latency.

      YMMV, though.

  7. Of course NT servers need this! by c.r.o.c.o · · Score: 2

    "as memory comprises 40 to 70 percent of the cost of most NT-based server configurations"

    That's because NT is bloatware. Now if everybody would run Linux, there would be no need for this technology, now would there..

    I'm sorry, but I just had to post this.

    1. Re:Of course NT servers need this! by spectecjr · · Score: 3

      "as memory comprises 40 to 70 percent of the cost of most NT-based server configurations"

      That's because NT is bloatware. Now if everybody would run Linux, there would be no need for this technology, now would there..

      I'm sorry, but I just had to post this.

      Actually, NT's not the problem. The problem is administrators who don't know how to take Exchange and SQL Server out of "Standalone" mode; as standard they pre-allocate a massive chunk of memory (as much as they can get; usually between 60-80%) so that they can run as fast as possible when they're on a dedicated box - which is the recommended way of setting them up on a large network.

      You can, however, turn this off. The registry key settings to do it are documented.

      Si

      --
      Coming soon - pyrogyra
  8. Stretching Old Tech Too Far by Digitalia · · Score: 2

    There is only so much that you can work with, before you need to start throwing out the old paradigm. Silicon is dying so countless engineers develop life support for it, while claiming that things are fine.

    Parallel computing, genetic algorithms, and this will not solve the problem. they will only prolong the world's suffering to a time when people will have become used to Big Silicon. Then all of the Wunderkinds of the Valley will be too old to execute for gross negligence.

    While this appears to be a good idea, we must understand that we can not grow too secure in silicon. When you get below .1 micron, you can't outrun the quantum boogeyman. Minimize, dis-consolidize, but realize that our future is not that of Silicon.

    --
    Pax Digitalia
    1. Re:Stretching Old Tech Too Far by MostlyHarmless · · Score: 2

      Ha ha. Then what do you suppose to replace silicon with?

      Look at it this way: If a technology existed that was good enough, people would use it. That's the way free market works.

      You could argue that the big manufacturers don't want to retool or are scared of the change, in the same way that the RIAA is scared to death of electronic music. In that case, upstarts with cost of retooling = $0 will take over with the new technology in the same way that Napster, Gnutella, and Freenet are taking over music distribution from the RIAA.

      --
      Friends don't let friends misuse the subjunctive.
  9. Real-world performance doubtful. by yakfacts · · Score: 2

    Any real-time compression technology tends to make me want to run screaming down the hallway

    This product does not promise to double your RAM, but "up to" doubling. Note that it says storage was doubled for "most applications".

    On MicroSloth machines where 3/4 of the memory seems to store arrays of zeros, this could be useful. But the more memory-concious the programmer, the poorer the performance of the technology. In other words, I'm not impressed.

  10. Double? Maybe yes, maybe no.... But great speed! by ucblockhead · · Score: 2

    With the disk compression utilities, one of the biggest troubles is that the amount that can fit on the disk suddenly depends on the type of data. You get amazing compression rates if you are story textfiles, horrible ones if you are storing GIFs. (The most common compressed file when disk compression was all the rage.)

    This will be similar. Suddenly the amount of memory an application uses is going to be less predictable. Perhaps this isn't much of an issue as with virtual memory, people are increasingly disconnected from application memory usage.

    Because of virtual memory, this is likely to greatly improve the apparent speed of the system, at least in cases where memory is moderately tight. (<128 Meg on a windows box, for example). Disk access is something like three orders of magnitude slower than memory access. If compression avoids even a few page faults, the lower page-file requirements will more than make up for the extra time to compress.

    --
    The cake is a pie
  11. thumbs down. by Signal+11 · · Score: 4

    Strike 1: "IBM Memory eXpansion Technology" BiCapitalization is the first sign of bad tech - it means the marketing people got to it before engineering could get it out the door. It also boils down to yet another meaningless TLA to impress PHBs: MXT.

    Strike 2: fake numbers. "as memory comprises 40 to 70 percent of the cost of most NT-based server configurations" Er, gee, not only is that an absurdly large error margin, but most servers cost, oh, we'll say $2000 and up. 40% of is $800. $800 of PC133 right now is about 640MB of RAM. Most systems in that price range have 256-384. Oops.

    Strike 3: Stating the obvious "and millions of tiny transistors" Oh, and how else would you do it? An analog circuit, perhaps?!

    Strike 4: Not promising: "The new technology is seamless to the end-user because the compressed data can be uncompressed in nanoseconds when needed." Call me a pessimist, but memory right now is around 6ns for PC133. Now, assuming a very conservative 2ns to decode the data, that's 8ns, which is a 25% performance hit. How many admins do you know that would take a 25% hit on performance on their servers to save a couple hundred bucks?

    In short, this new tech is gonna tank.

    1. Re:thumbs down. by ucblockhead · · Score: 3

      . $800 of PC133 right now is about 640MB of RAM.

      Not all memory is in the DIMMS. There are caches everywhere.

      Now, assuming a very conservative 2ns to decode the data, that's 8ns, which is a 25% performance hit.

      You are forgetting what happens when you run out of memory. You page fault, and have to access the block off of disk, which takes something like 9 milliseconds. Getting rid of one page fault for every 1000 memory accesses is going to completely wipe out that 25% performance hit.

      In other words, if you are using any sort of paging file, this will almost certainly improve performance, not hurt it.

      --
      The cake is a pie
    2. Re:thumbs down. by supabeast! · · Score: 2

      "NT-based server configurations"
      "but most servers cost, oh, we'll say $2000 and up"

      What the HELL kind of servers would those be? Quake servers? $2000 might make a decent Linux server, but $2000 is barely a high end Intel based desktop, much less a server. Good NT servers (at least NT servers that handle a large volume of, well, just about anything) cost two to three times that and have at least a gigabyte of RAM. Now that many Intel server buyers are stuck buying RDRAM to get around the horrible problems of SDRAM on an RDRAM mobo, RAM certainly does eat up that much of the cost.

    3. Re:thumbs down. by cperciva · · Score: 2

      Strike 2: fake numbers. "as memory comprises 40 to 70 percent of the cost of most NT-based server configurations" Er, gee, not only is that an absurdly large error margin, but most servers cost, oh, we'll say $2000 and up. 40% of is $800. $800 of PC133 right now is about 640MB of RAM. Most systems in that price range have 256-384. Oops.

      It isn't an NT server, but a 128 processor Beowulf was recently built at the university of British Columbia, and memory comprised 44% of the total cost.

      I don't think that 40-70% is unreasonable at all.

      --
      Tarsnap: Online backups for the truly paranoid
  12. Re:Does that mean software compression is now moot by yakfacts · · Score: 2

    Software compression in RAM, or on disk? Software compression on disk is, of course, unchanged. Compressed data structures in RAM will now bloat...so you will be penalized for using them. That is why I hate forced compression.

  13. I can't wait for the advertising by SIGFPE · · Score: 2

    Remember how for about a year or so manufacturers advertised that they were selling computers with 200Mb hard drives but when you looked in the small print it said "With XXX installed" where XXX was some disk 'doubler'? I guess we're going to get a year of companies misleadingly trying to sell 1/2 Gigabyte PCs claiming they're Gigabyte PCs. You have been warned!
    --

    --
    -- SIGFPE
    1. Re:I can't wait for the advertising by Sloppy · · Score: 2

      It's already standard practice with tape drives, no need to remember way-back-when. Every single tape drive on the market is advertised as having twice as much capacity as it really has. (One of my pet peeves.) Yes, it would suck if that happened to the RAM market too.


      ---
      --
      As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
    2. Re:I can't wait for the advertising by Kris_J · · Score: 2
      Tape drives are advertised with two numbers - uncompress/compress, ie; 4/8 for DDS2 or 12/24 for DDS3. This tells you that there is a hardware (or simply transparent) data compression system somewhere in the pipeline. I typically expect a 1.5x increase on data of Office97 or later vintage (Word and Powerpoint 97 save images compressed, Pre-97 versions save images uncompressed). Thus I would consider a 4/8 to give me about 6Gig, a 12/24 to give about 18Gig.

      If there was a similarly transparent (including no noticable performance hit) RAM compression technology built into a system or a SIMM then I wouldn't object to a 64/128 or similar description.

      However, advertising just the compressed size is immoral and deceptive...

  14. Re:This is not NEW!! by exploder · · Score: 2

    What's so new is that it's implemented in hardware. The article claims approximately 4 orders of magnitude better performance than a software solution. Of course, whether that's enough to make it worthwhile is not addressed.

    --
    Yo dawg, I heard you like the Ackermann function, so OH GOD OH GOD OH GOD
  15. Latency vs. Bandwidth by Guppy · · Score: 3

    "This would rock on the PC, where our bus speeds are slowly reaching 133mhz. If you could send the data compressed across the bus, that is."

    Maybe, but I'll bet there will be a fairly large impact on latency, with the overhead needed for compression/decompression. Bandwidth is more important for some kinds of memory-intensive applications, like database and photo/video editing stuff. But for everyday applications or games like Q3, the latency is going to knock your performance way down. It's one of the problems with Rambus, where the bandwidth improves but latency actually gets worse.

  16. Re:thumbs down. (Maybe not) by exploder · · Score: 2

    Call me a pessimist, but memory right now is around 6ns for PC133. Now, assuming a very conservative 2ns to decode the data, that's 8ns, which is a 25% performance hit.

    Well, let's say that the stated 2:1 compression ratio is acheived. Now we're moving twice as much data in 133% of the time, which is a 33% performance gain. (2x the data in 8ns, equivalent to same data in 4ns, as compared to the original 6ns.) The break-even point for 2:1 compression is a 2:1 slowdown in performance. If the compression averages 1.5:1, then the performance must be no worse than 1.5x as slow in order to avoid degrading access time. Can the average performance cost ratio go below the average compression gain ratio, and actually increase performance? If not, then how close is tolerable? I'd say there's room for this technology to be of value, especially as stated in the article, in servers with enormous amounts of RAM.

    --
    Yo dawg, I heard you like the Ackermann function, so OH GOD OH GOD OH GOD
  17. Speed Hit? by Dungeon+Dweller · · Score: 2

    I use RAM because it's fast, if we didn't care about speed, we would all be writting to our hard drives and have about 2 megs of RAM and just use swap partitions for all of our memory needs. This won't last because, no matter how fast it is, it will be faster to write straight to the RAM.

    --
    Eh...
  18. Re:This is not NEW!! by X · · Score: 2

    Ok. Even if it's done in hardware this kind of thing has been around for quite some time. The Hobbit processors do this, along with a slew of other embedded processors.

    One thing to think about is how much faster is a hardware implementation really? Time and time again general purpose CPU's seem to kick the butts of dedicated hardware in all but the most esoteric cases (like encryption). If it's done by the CPU then the data you need is already in the L1 cache and possibly in registers, all while avoiding pain for the outer caches. Then add in architectures like EPIC which have nothing better to do with some of their units anyway...

    I don't know, I don't know if IBM's claims will pan out.

    --
    sigs are a waste of space
  19. A small test by PD · · Score: 3

    I took a snippet from /proc/kcore and compressed it to see how well it would work.

    dd if=/proc/kcore of=/spare/mem bs=1 count=1000000

    that resulted in a chunk of kcore 1 million bytes long written to the file /spare/mem

    bzip2 -9 /spare/mem

    that resulted in a file named mem.bz2 being 349791 bytes long.

    This was on your typical RedHat system running the usual stuff.

    --
    If tits were wings it'd be flying around.
  20. interesting, maybe. by Phexro · · Score: 2
    i recall the days when you could get software to solve any number of hardware problems. not enough ram? get ramdoubler. not enough disk space? get stacker. computer not fast enough? get 386to486.exe!

    a rule-of-thumb i recall from that era was something along the lines of: 'software solutions to hardware problems are impractical'.

    the fact that they do the compression in hardware may have some merit. so i did a bit of testing; i checked the sizes of /proc/kcore, and the size after piping /proc/kcore through gzip and into a file.

    on my 32mb box: (4944k used, not counting cache)

    compressed uncompr. ratio uncompressed_name
    18796861 33558528 43.9% kcore


    on my 192mb box: (144872k used, not counting cache)

    compressed uncompr. ratio uncompressed_name
    99302828 201265152 50.6% kcore


    figures are probably quite skewed, since the core image was not a snapshot. but it looks like the actual used memory compresses better then the bit-soup that is in the dimms when the system powers up.

    who knows... maybe ibm has a few tricks up their sleeves. be interesting to see some linux source to deal with these beasts... i'm assuming that it's os-dependent, and since ibm has been great about linux lately, i'd think they would release whatever kernel patches would be necessary to use these things.

    --

  21. Preventing memory expansion; cache block sizes by billstewart · · Score: 2
    A standard hack that on-the-fly compressors use is to keep a flag bit indicating whether a given chunk is compressed or uncompressed. That way you can avoid the problem that recompressing already-compressed data often grows, at the cost of a flag. On the other hand, if most of your data is compressed anyway, like web servers caching lots of JPEGs, you still don't win, and you're hauling it uncompressed on most of the busses anyway.

    I am interested in how big a block of memory this stuff operates on. Compressing a few K at a time, such as a disk block, may be big enough to win, but compressing a 128-bit cache line almost certainly loses. Where's the breakeven point? The hope of this technology is that if you stick it way out in L3 cache, you're usually hauling big enough chunks at a time that the decompression latency for getting the first byte is made up for by the bandwidth of getting the rest of the bytes from a smaller amount of memory.


    Help, help, I'm being compressed!

    --

    Bill Stewart
    New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
  22. What this actually is... by ucblockhead · · Score: 5

    Is a caching system. From the article:

    MXT is a hardware implementation that automatically stores frequently accessed data and instructions close to a computer's microprocessors so they can be accessed immediately -- significantly improving performance. Less frequently accessed data and instructions are compressed and stored in memory instead of on a disk -- increasing memory capacity by a factor of two or more.

    Note two things: They are not compressing everything. They are not replacing the actual memory.

    Most of the criticisms here are based on misunderstandings of those two things.

    (Note that I'm guiltless. I posted a number of times before getting around to read it.)

    --
    The cake is a pie
  23. 3:1 a bit optimistic by Wesley+Felter · · Score: 2

    RAM compression isn't going to deliver compression ratios as good as stream oriented algorithms like deflate and bzip2, because it has to be random-access and the compressor doesn't have as much context to look at.

    But as you pointed out, memory tends to be pretty compressible because of the significant redundancy.

  24. Re:BIOS level support by drinkypoo · · Score: 2

    Since this is hardware located on the motherboard, don't look for it in the desktop too soon, unless Rambus licensing fees drive it.

    Since it is on the motherboard it would probably require support at the bios level. Also because it has its own caching system, it may only work with certain CPU chips, maybe, maybe not.

    The real question is, where does this technology live? Is it in the North Bridge, or in some other bizarre location? Or is it on the DIMM?

    See, for it to be truly transparent (Which it more or less has to be) it's going to have to be in the chipset, or on the DIMM. Putting it on the DIMM means there has to be one of these suckers for each DIMM, but then that might be true already. It seems to make the most sense to embed it into the chipset, or to sandwich it between the chipset and the memory.

    If IBM does it right, then that means that you will not need any OS support for this "technology". From the article given, it doesn't really sound like there's anything amazingly new here, but that remains to be seen. I'd want to see the patent involved before I made any calls on this. In any case, hardware data compression has been around for a long time. Heck, take as a [lousy] example the FWB Jackhammer NuBus SCSI cards for the Mac. Those did compression in hardware, though they did have a software component (the driver written to the hard disk.) I'm envisioning this as a more transparent way of handling it.

    It's not unreasonable to say that IBM could get 2:1 compression ratios reliably with a good algorithm and some high-speed logic. It's even likely that it will be cheaper to buy 512mb using this technology than 1gb without it; It's even more likely that it'll be cheaper to buy 2gb with it than 4gb without it, because the larger the DIMM, the higher the cost per megabyte, once you get past a certain point (about 32mb.)

    Anyway, show me the patent. I'd like to read it.

    --
    "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  25. Re:It won't last (it will if they keep it updated) by Kris_J · · Score: 2
    The reason I stopped using Drivespace(3) was that MS never made a FAT32 compatible version, not because I decided that it sucked.

    Bias Disclaimer: I think that predominantly transparent compression of a large storage area rocks. I'm also a huge fan of decicated hardware accelleration. Thus I think this is the sort of cool idea I'd want regardless of whether or not it actually works.

    That said, this should provide some performance increase to any system with a swap file, like my little portable (not that it can be retro-fitted) - but I would like to know how the OS will deal with a variable memory size...

  26. Re:This is not NEW!! by Kris_J · · Score: 2
    Time and time again general purpose CPU's seem to kick the butts of dedicated hardware
    Only if that's the only thing they're doing. Most of my portable's CPU time is taken up playing MP3s. If I could farm that task off to dedicated hardware then I'd have more CPU time for whatever, disk compression maybe (or D.net or SETI@home ;).

    Also, I'd tend to think that the dedicated hardware accelleration in today's video cards is not especially esoteric...

  27. It's all about cost by Kris_J · · Score: 2
    A powerful general purpose CPU is all well and good, but they're expensive (and power hungry). If, for some specific tasks, a much cheaper bit of dedicated hardware can perform just as well then makes sense to farm off the task to that cheaper hardware.

    A good way to get a quick feel for this is to look at portable MP3 players v's playing MP3s on a WinCE/PocketPC unit. A dedicated MP3 player gives you 64-96MB of RAM for around US$300. A WinCE/PPC unit that can play MP3s will typically cost US$600-$1000 and only include 16MB-32MB. And the battery life is 12hours v's 3 or less.

  28. Re:Latency will not be affected... by be-fan · · Score: 2

    This really has nothing to do with disk caching, it compresses memory, not disk space. And it WILL increase latency because a section of memory must be decompressed before it can be sent. Even with dedicated hardware it will still take a few clocks.

    --
    A deep unwavering belief is a sure sign you're missing something...
  29. Re:NVidea is already doing this with graphics card by be-fan · · Score: 2

    It's not just the new drivers from nVidia. What nVidia does is DXTC (DirectX texture compression) which is a form of S3TC (S3 texture compression.) 3DFx is doing this to with FXT1 (I think that's right)

    --
    A deep unwavering belief is a sure sign you're missing something...
  30. Re:Latency will not be affected... by be-fan · · Score: 2

    Opps. Disregard my previous comment. I was listening to the topic instead of reading the article. You're right about everything. I'm sorry. Forgive me. ;)

    --
    A deep unwavering belief is a sure sign you're missing something...
  31. Here is how to hit it in real life by tilly · · Score: 2

    mmap() a large encrypted file. Start decrypting.

    When you do that in the background the OS will have to start paging in the encrypted file. It will stay in memory until that memory is needed for something else. But the encrypted file is a worst case scenario for the compression algorithm.

    If someone has an encrypted filesystem this will actually be an extremely common case!

    Really, your "Oh we will never hit that in the real world" is exactly how programmers f*ck up time after time again. You may not see how it will happen, but it will happen eventually and people will get hosed.

    Deal with your corner cases before turning your code lose on the world, please.

    Regards,
    Ben

    --
    My usual seat in the cluetrain is at A HREF="http://pub4.ezboard.com/biwethey.ht
  32. Disk compression by Shotgun · · Score: 2

    One of the arguments for using software disk compression back in the day was that it would increase transfer rates. It was faster to move less data across the disk bus and then process it with a fast processor than it was to move the uncompressed data directly to memory.

    1) How true was this claim, and if it was true, why isn't it still true?

    2) Will the same effect be apparent in harware memory compression? Furthermore, would more performance be seen if the compressor was moved onto the processor so that not as much data had to be moved across the memory bus?

    --
    Aah, change is good. -- Rafiki
    Yeah, but it ain't easy. -- Simba
  33. Cache technology? by jovlinger · · Score: 2

    yeah, what he said!

    I haven't had a look at the press release, only the AP article. The one place this would be useful is in the 2nd or 3rd level cache. If you can compress fast enough then the extra size of the cache would come in handy.

    The reason why this technique would work on a cache but I can't see how it would work on main memory is that you don't ever see the cache, so you never make assumptions about how large it will be. So the encrypted data sitting in the cache fills it up after 1 Meg, while the empty matrix full of zero can have almost 4 Megs cached.

    Cache is transparent, so you can never be bitten by it changing logical size on you. Unlike main memory.

    For main memory, like sig11 points out, the OS will make assumptions that it can store exactly that much info. And since the hardware has no provisions for asking the OS to page out memory (and has no business asking, either) eventually havock will be wrought.

    So I'm strongly suspecting that this will turn out to be used for 3rd or 4th level cache.