Anatomy of Linux Kernel Shared Memory

← Back to Stories (view on slashdot.org)

Anatomy of Linux Kernel Shared Memory

Posted by kdawson on Saturday April 17, 2010 @08:51AM from the culture-by-another-name dept.

An anonymous reader sends in an IBM DeveloperWorks backgrounder on Kernel Shared Memory in the 2.6.32 Linux kernel. KSM allows the hypervisor to increase the number of concurrent virtual machines by consolidating identical memory pages. The article covers the ideas behind KSM (such as storage de-duplication), its implementation, and how you manage it.

93 comments

First Post by stink_eye · 2010-04-17 09:06 · Score: 1, Redundant

Seems to me this feature has been available for a while in VMWare...
1. Re:First Post by Anonymous Coward · 2010-04-17 09:15 · Score: 2, Informative
  
  VMWare? Fuck, this has been around for decades in the case of OS/360.
2. Re:First Post by Anonymous Coward · 2010-04-17 09:27 · Score: 5, Informative
  
  From the article:
  "Going further
  Linux is not alone in using page sharing to improve memory efficiency, but it is unique in its implementation as an operating system feature. VMware's ESX server hypervisor provides this feature under the name Transparent Page Sharing (TPS), while XEN calls it Memory CoW. But whatever the name or implementation, the feature provides better memory utilization, allowing the operating system (or hypervisor, in the case of KVM) to over-commit memory to support greater numbers of applications or VMs. You can find KSM—and many other interesting features—in the latest 2.6.32 Linux kernel."
3. Re:First Post by BitZtream · 2010-04-17 09:27 · Score: 1, Interesting
  
  The concept isn't new to Windows, VMWare or FreeBSD I know for a fact (though none of them work exactly the same as this).
  I would have presumed this wasn't new to Linux either, just different from the existing implementation (I know its blasphemy here, but I'm not a Linux person).
  Its certainly been done in the mainframes for god knows how long.
  I doubt this is as groundbreaking as its being promoted.
  If your OS isn't sharing duplicate memory blocks already, you're using a shitty OS. (Linux already shares dup read only blocks for many things, like most modern OSes).
  These are just extensions to help deal with virtual machines rather than actually fix the problems that make the need to run a virtual machine. Not unique to Linux in any way, all the big boys are riding the VM fad until people realize your OS is supposed to do that for you already. Prolly take a few years before all the young'ns realize that VMs aren't new and theres a reason people having been using them all the time for the last 3 decades (at least, my age is showing, but the 70s era computing was just before I started :)
  
  --
  Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
4. Re:First Post by Abcd1234 · 2010-04-17 09:34 · Score: 5, Insightful
  
  If your OS isn't sharing duplicate memory blocks already, you're using a shitty OS. (Linux already shares dup read only blocks for many things, like most modern OSes).
  Umm, no.
  Most modern OSes share memory for executable images and shared libraries. In addition, some OSes, such as Linux, support copy-on-write semantics for memory pages in child processes created with fork (note, Solaris is an example of an OS that *doesn't* do this).
  Aside from that, there is no automated sharing of memory between processes. Frankly, I have no idea where you got the idea there was.
5. Re:First Post by Anpheus · 2010-04-17 10:24 · Score: 3, Interesting
  
  For now, at least. VMWare doesn't support combining pages >= 2MB because the overhead (hit rate on finding duplicates versus the cost of searching for duplicates) and I suspect other hypervisors will do the same. Additionally, Intel and AMD are both moving to support 1GB page tables. What are the odds that you'll start up two VMs and their first 1GB of memory will remain identical for very long?
  The only way I see page sharing working in the future is if the hypervisor inspects the nested pages down to the VM level, which will typically be the 4KB pages we know and love. Either that, or paravirtualization support needs to exist for loading common code and objects into a shared pool.
  Even so, there's a lot of overhead from inspecting (hashing and then comparing) pages which will only grow as memory sizes grow. If we increase page sizes, the hit rate decreases and the overhead of copy-on-write increases. It's not a good situation.
  Sources: Performance Best Practices for vSphere 4 which references Large Page Performance which states:
  
  In ESX Server 3.5 and ESX Server 3i v3.5, large pages cannot be shared as copyonwrite pages. This means, the ESX Server page sharing technique might share less memory when large pages are used instead of small pages. In order to recover from nonsharable large pages, ESX Server uses a “sharebeforeswap” technique. When free machine memory is low and before swapping happens, the ESX Server kernel attempts to share identical small pages even if they are parts of large pages. As a result, the candidate large pages on the host machine are broken into small pages. In rare cases, you might experience performance issues with large pages. If this happens, you can disable large page support for the entire ESX Server host or for the individual virtual machine.
  That is, page sharing involves breaking up large pages, negating their performance benefit and is only used as a last ditch when you've overcommited memory and you're nearly to the point of having to hit the disk. And VMWare overcommit is great until you hit the disk, then it's a nightmare.
6. Re:First Post by aBaldrich · 2010-04-17 10:39 · Score: 1
  
  You are right, the "news" here, is that someone made an article explaining KSM, like, slashvertising.
  
  --
  In soviet russia the government regulates the companies.
7. Re:First Post by keeboo · 2010-04-17 12:37 · Score: 1
  
  In addition, some OSes, such as Linux, support copy-on-write semantics for memory pages in child processes created with fork (note, Solaris is an example of an OS that *doesn't* do this).
  Wait, Solaris doesn't do copy-on-write? (uh-oh)
  I, too, expected that to be pretty much standard nowadays.
8. Re:First Post by Trepidity · 2010-04-17 14:04 · Score: 2, Insightful
  
  This article claims that on Solaris, "fork() has been improved over the years to use the COW (copy-on-write) semantics". It's sort of an in-passing comment, though, and I can't find a definitive statement in docs anywhere (the Solaris fork() manpage doesn't give any details).
  
  --
  10 PRINT CHR$(205.5+RND(1)); : GOTO 10
9. Re:First Post by bzipitidoo · 2010-04-17 14:09 · Score: 2, Interesting
  
  Personally, I find memory compression more interesting than just deduplication, which could be considered a subset of compression. The idea has been around for years. There used to be a commercial product for Windows 95 that claimed to compress the contents of RAM, but which had many serious problems, such as that the software didn't work, didn't even attempt compression, and was basically "fraudware". The main problem with the idea was it proved extremely difficult for a compression algorithm to beat the speed of just doing everything without compression. Gave the idea of memory compression a bad reputation.
  Now we have LZO, an algorithm that has relatively poor compression, but which is very, very fast-- fast enough to beat the speed of a straight memcopy. 15 years ago, there wasn't any compression algorithm fast enough for this application. Also, I'm thinking memory is slower relative to processors than 15 years ago, as that would provide incentive to increase the size and sophistication of CPU caches, which has happened. (100 MHz, 32bit Pentium with 33 MHz RAM then, vs 3 GHz, 64 bit multi core CPUs with 800 MHz DDR2 RAM today) Now CPU caches are plenty large enough to handle many 4K pages.
  Still, deduplication could have many other cool uses. Friend of mine once hacked up some disk deduplication software for the Apple II. All it did was crosslink identical sectors it found on a disk, and then mark the duplicates as free. There was no provision for counting the number of links or anything like that, in case you wanted to change the contents of the disk. Just had to be aware this had been done to the disk. Ultimately proved more trouble than it was worth, but it was a nice thought for teenagers desperate for more disk space.
  
  --
  Intellectual Property is a monopolistic, selfish, and defective concept. It is "tyranny over the mind of man"
10. Re:First Post by Anonymous Coward · 2010-04-17 15:20 · Score: 0
  
  That article is four years old, and CoW wasn't new then either.
  Linux doesn't even ship with fibre channel drivers!
  See what I did?
11. Re:First Post by IkeTo · 2010-04-17 15:33 · Score: 1
  
  If your OS isn't sharing duplicate memory blocks already, you're using a shitty OS. (Linux already shares dup read only blocks for many things, like most modern OSes).
  That depends on how the memory gets duplicated. If it is duplicated because it comes from the same library or because it is the result of forking a process, you're right, every OS does that. But if it is because the memory content comes from independent processes doing independent things and the result happens to be exactly the same, it is new. In the former case, if two processes are sharing some memory, one process decide to write over it, and the other does exactly the same write, then the result is two unshared pages. In the latter case, they will also become unshared at that moment, but after a while the OS will notice that they are the same again and make them share again.
  I think that there is not a lot of use cases of KSM actually apart from the VM use case. That's why nearly no OS provide that service until today. And even if Linux provide that, it is likely that most people running a Linux desktop will not enable the feature because it wastes clock cycles to ask the OS to constantly scan through your memory to find pages to deduplicate.
12. Re:First Post by MichaelSmith · 2010-04-17 15:38 · Score: 1
  
  Now we have LZO, an algorithm that has relatively poor compression
  No kidding. We compress trace files with LZO at my day job and the compressed versions still have large human readable chunks.
  
  --
  http://michaelsmith.id.au
13. Re:First Post by BitZtream · 2010-04-17 15:49 · Score: 0, Troll
  
  So if you disagree, why did you go ahead and point out examples for me?
  Aside from all the places that memory is shared between processes, theres no sharing between processes ... yea, I totally get you ...
  Yes, if you exclude everything else, this is new and exciting.
  
  --
  Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
14. Re:First Post by ToasterMonkey · 2010-04-17 15:58 · Score: 1
  
  In addition, some OSes, such as Linux, support copy-on-write semantics for memory pages in child processes created with fork (note, Solaris is an example of an OS that *doesn't* do this).
  What year do you live in? Solaris _9_ had COW and multiple page size support, over half a decade ago. Linux large page size support is a joke, Solaris x86 even does it better on Linux's home turf.
  http://www.sun.com/blueprints/0304/817-5917.pdf
  Most modern OSes have a native fibre channel stack, except notably, Linux which doesn't have userland utilities for managing SCSI devices or even fibre channel drivers for that matter.
  See what I did?
15. Re:First Post by Abcd1234 · 2010-04-17 17:28 · Score: 1
  
  What year do you live in? Solaris _9_ had COW and multiple page size support, over half a decade ago.
  *snicker* I like how you phrased that as "half a decade ago"... "five years ago" sounds far less impressive, when you consider how industrial strength Solaris has traditionally been considered. :) That said, I fully concede my information is probably out-of-date. Glad to see Solaris finally moved into the 21st century!
  Besides, it wasn't a criticism of Solaris (the only reason I came across the factoid at all was because I was doing work with very large numbers of processes on a single box, and Linux massively outperformed Solaris at the time, partly because of Linux's COW semantics for forked processes). My point was simply that memory sharing is the exception rather than the rule for OSes, in that there is a very limited scope in which it applies.
  Seriously, sensitive much?
16. Re:First Post by Abcd1234 · 2010-04-17 17:33 · Score: 4, Insightful
  
  Aside from all the places that memory is shared between processes, theres no sharing between processes ... yea, I totally get you ...
  That's exactly right. I pointed out all the places. *All of them*. And there's *two*: shared, read-only executable pages, and the heaps of children created by COW-enabled forks. That's it. That's all.
  So any new technology for memory de-duping is impressive because, traditionally, it just ain't done. Which directly contradicts the content of your original post.
  Perhaps now you understand?
17. Re:First Post by shadanan · 2010-04-17 20:13 · Score: 1
  
  That's because the LZO algorithm only attempts compression on data with runs of the same repeating data, and data that would match multiple instances of a sliding dictionary. Data that doesn't satisfy these conditions within a given block are not compressed.
18. Re:First Post by improfane · 2010-04-17 20:45 · Score: 1
  
  The breakthrough nature of this is that the hypervisor or the host OS is providing a virtual machine to every guest OS in the system. A virtual machine provides an environment that mirrors the real hardware, the OS knows no better. This in theory means that you could run multiple Linux distributions with the memory of a Linux kernel only being used once, meaning more applications can be run within these guest OSes or more guest Oses.
  That's why it is impressive.
  
  --
  Slashdot needs Geekcode | Can anyone recommend any good SCIFI? My tastes: Foundation, Startide Rising, CITY, Ringworld,
19. Re:First Post by Anonymous Coward · 2010-04-17 20:46 · Score: 0
  
  And this is why Hyper-V doesn't bother to implement page sharing at all - the argument runs you should be using large pages, and the trend to using large pages will only increase. With large pages, the chances of having bit-for-bit identical pages is too low to make the overhead of scanning for such pages to be worthwhile.
  http://blogs.technet.com/virtualization/archive/2010/04/07/dynamic-memory-coming-to-hyper-v-part-3.aspx
20. Re:First Post by nabsltd · 2010-04-18 02:00 · Score: 2, Informative
  
  So any new technology for memory de-duping is impressive because, traditionally, it just ain't done. Which directly contradicts the content of your original post.
  For those who are still confused, the big difference between the various shared library-type schemes and memory de-dupilication is passive vs. active.
  Shared libraries (or executables) take advantage of the fact that when you load an program multiple times, the same bits are obviously being loaded each time and so it's just a reference count increment.
  For memory de-duplication, during idle times, the hypervisor creates hashes of all the used memory pages and if any duplicates are found they are replaced with the same sort of reference count as in the shared library approach. This would allow me to do something like copy the "vi" command to my home directory and rename it "edit", but the system will figure out that it's the same pages as the real "vi", so those pages will only be in memory once for all users.
  As far as I know, only hypervisors use the active memory de-duplication approach...no regular OS does what I suggested in my "vi" example.
21. Re:First Post by Anonymous Coward · 2010-04-18 02:04 · Score: 0
  
  Interesting point...Microsoft decided not to implement page sharing in the latest rev of Hyper-V because operating systems are moving from 4k pages to large pages (seems to me that they were 2MB in size). When dealing with larger page sizes the probability of finding shareable (aka duplicate) pages is incredibly slim, with the exception of zeroes pages. So page sharing ends up being of very little benefit.
  The other point is that modern operating systems are now making more full use of the memory allocated to them, so that even if an application does not require the full amount of memory available the OS will use that memory space for caching to speed up OS operations (as opposed to pulling data from disk), thereby reducing the number of zeroed pages and making page sharing an even more marginal technology.
  Just a thought...
22. Re:First Post by x2A · 2010-04-18 16:14 · Score: 1
  
  Be interesting to see actual speed differences of favouring larger page sizes vs fewer duplicate pages... on the one side, you get fewer TLB misses which slows things down, but on the other side, you can - in effect - be increasing your L2 cache, depending on scheduling. By which I mean that if you have a page that's shared between three processes (VMs or otherwise) then any cachelines covering data within that space effectively covers three times the memory that it would have to cover if they weren't shared. I guess, as with anything, that there are probably workloads where each alternative performs better than the other.
  
  --
  The revolution will not be televised... but it will have a page on Wikipedia
23. Re:First Post by Anonymous Coward · 2010-04-19 00:28 · Score: 0
  
  So page sharing ends up being of very little benefit.
  It does? Always? Or some times, under certain circumstances?
  How about the case with a "large"-ish machine running 100+ instances of the same virtual system image? Would there still be very little benefit?
  Do you have any references to information corroborating the Microsoft stance on this issue?
  I'd certainly be interested in reading some of it.
24. Re:First Post by veliath · 2010-04-19 09:18 · Score: 1
  
  Solaris is an example of an OS that *doesn't* do this
  Are you sure? When the child modifies its globals or anything in its heap, the memory has to be COWed.
25. Re:First Post by Anonymous Coward · 2010-04-26 03:49 · Score: 0
  
  solaris sucks you suck enjoy your sinking ship of an OS as it finally glides into that faithful iceberg and oracle shits on your childhood memories.
i love these by mehemiah · 2010-04-17 09:08 · Score: 1

they are so informative now that the KernelTrap isn't updating regularly ... AHEM! *cough! *cough! there is some reorganization of linux graphics and networking that needs to be updated. (or has linux networking changed sense 2005
1. Re:i love these by ISoldat53 · 2010-04-17 11:05 · Score: 1
  
  You should take something for that cough.
2. Re:i love these by Runaway1956 · 2010-04-17 12:47 · Score: 0, Troll
  
  "or has linux networking changed sense 2005"
  No, Mr. Ballmer - nothing has changed in Linux in years now. It's safe to stick your head back up your arse, and assume that Windows is superior to everything in the world. The schmucks out there still believe it, and sales are stable.
  
  --
  "Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
This Is Just One Reason ... by WrongSizeGlass · 2010-04-17 09:09 · Score: 4, Interesting

... why OSS is the way things should be. You'll never see this type of documentation, and this type of detail, available to anyone and everyone from closed source software. I love my Mac, and supporting Windows pays my bills, but OSS is unlike any other animal out there.
1. Re:This Is Just One Reason ... by siride · 2010-04-17 09:19 · Score: 0
  
  There are books detailing the NT Kernel and the OS X kernel (which is open source, after all).
2. Re:This Is Just One Reason ... by BitZtream · 2010-04-17 09:20 · Score: 0
  
  Then I take it you've never bothered to look at msdn.microsoft.com or developer.apple.com in depth have you?
  Contrary to popular belief, most of us who actually have a clue have been dealing with far better documentation from MS and Apple than Linux has ever had. And no, the source doesn't count if no one knows what you intended to do, which is effectively what you get from it.
  Get a clue fanboy.
  
  --
  Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
3. Re:This Is Just One Reason ... by abigor · 2010-04-17 09:41 · Score: 4, Informative
  
  OS X's kernel is open source (BSD license) and very well documented.
4. Re:This Is Just One Reason ... by Anonymous Coward · 2010-04-17 09:56 · Score: 0
  
  anyone who thinks msdn is actually well laid out and easy to read must be a fanboy...or an idiot. apple is worse because they refuse to use standard terms for the tech they use. for ex, try getting technical details about quicktime's encode profiles. good luck.
  even if 'those who have a clue' have signed some kind of license or NDA to get access to a level of info akin to what's offered by many OSS projects, that doesn't help the average tech who needs to get those vendors' products working correctly. with OSS, the documentation is a few clicks away with no hidden apis and/or legal threats for using them.
  get a clue fanboy.
5. Re:This Is Just One Reason ... by Anonymous Coward · 2010-04-17 09:58 · Score: 0
  
  I never understand why clueless people resort to making up facts to defend their beliefs/leaders/country/software. So maybe Linux is the better OS. It doesn't mean everything up to and including the documentation is superior. A budding systems programmer can learn just about everything he needs to know by reading MSDN. A newbie Linux programmer is pretty much fucked if he doesn't have years of prior experience.
6. Re:This Is Just One Reason ... by Anonymous Coward · 2010-04-17 10:02 · Score: 0
  
  Another person who confuses programmer complaints of MSDN API documentation and MSDN OS documentation. Don't bother writing about MSDN if you have never used it yourself.
7. Re:This Is Just One Reason ... by icebraining · 2010-04-17 11:09 · Score: 1
  
  What OS are your using? It's not Mac OS X nor Windows, both have OSS components.
  
  --
  Dilbert RSS feed
8. Re:This Is Just One Reason ... by mswhippingboy · 2010-04-17 12:18 · Score: 3, Insightful
  
  far better documentation from MS and Apple than Linux has ever had.
  Have you ever looked at Amazon or InformIT/Safari or any technical documentation vendor or website? There are enough books and articles on MS and Linux to keep you reading for many lifetimes (Apple not so much, but still plenty by my estimate). It's just a FACT that there are certain things that closed source vendors do not disclose as a matter of trade secret or intellectual property, which is what I believe WrongSizeGlass was referring to. OSS does not have this limitation.
  
  And no, the source doesn't count if no one knows what you intended to do
  It absolutely does count, if you know how to read code. You can read documentation all day long, but if it doesn't match the code you'll be lost.
  When I get involved in a rewrite/upgrade/modification of an existing application (and I've worked on many much larger than most OSS apps, and I've been doing it for over 30 years - so it think I have a clue), I read the docs for a quick overview, but I read the code to find out how it really works.
  I've spent plenty of time with both msdn.microsoft.com (and developer.apple.com to a lesser degree) over the years and while there are loads of information there, I know "everything" is not there. There are API documentation discrepancies, quirks, or various undocumented oddities that can be frustrating to work around. These issues can only be resolved one of two ways - trial and error or by reading the code.
  With closed source there is only one way - pray the docs are correct. With open source there are usually two ways and if the documentation is shitty, you can be sure the code is correct. That makes my life as a developer easier.
  
  Get a clue fanboy.
  Wow. I'm sure you won him over with that jab. First an outpouring of pomposity and narcissism, then a put-down to finish it off.
  Good job dude.
  
  --
  Sometimes the light at the end of the tunnel is the headlight of an oncoming train.
9. Re:This Is Just One Reason ... by mswhippingboy · 2010-04-17 12:30 · Score: 1
  
  I never understand why clueless people resort to making up facts to defend their beliefs/leaders/country/software
  Physician heal thyself.
  
  A newbie Linux programmer is pretty much fucked if he doesn't have years of prior experience.
  A newbie Linux programmer is not likely to be mucking around in the kernel now is he?
  
  --
  Sometimes the light at the end of the tunnel is the headlight of an oncoming train.
10. Re:This Is Just One Reason ... by c4t3y3 · 2010-04-17 13:37 · Score: 1
  
  And no, the source doesn't count if no one knows what you intended to do
  
  It absolutely does count, if you know how to read code.
  
  Common sense please, no matter your expertise in programming, you can't understand some code unless documented. Example: the X Server.
11. Re:This Is Just One Reason ... by mswhippingboy · 2010-04-17 15:03 · Score: 1
  
  I really don't care to debate the point. While it may be difficult, and I've seen some pretty horrendous code, if it compiles, it's understandable. That's common sense. Computer languages have very precise grammers and semantics.
  In all my years, I've never once told my clients "I can't understand it. It's just too hard". Consider yourself lucky if you have the code. I've even had to reverse engineer code from binaries on occasion. That's hard, but again, not impossible because if the CPU can execute it, it's understandable.
  And for the record, I worked on some custom mods to Xserver (xfree86) and I didn't find them any more difficult to understand than much of the C code I've seen. In fact, while by it's nature it's very complex software, I thought the code was well written for the most part. Of couse, you have to be comfortable with C (conditional compilation, macros and pointers, etc), as well as the concepts being implemented, but the concepts are fairly well documented on the Internet and in several books.
  
  --
  Sometimes the light at the end of the tunnel is the headlight of an oncoming train.
12. Re:This Is Just One Reason ... by Saint+Stephen · 2010-04-17 15:35 · Score: 2, Insightful
  
  Take it from a guy who's seen the NT source code: Inside Windows 2000, the windows kernel debuggers, and a firewire cable gave you MORE than enough detail; there's not much important that's not publicly known.
  It just doesn't make Slashdot or the sites you frequent. How do you think Windows Device Driver writers do their job?
13. Re:This Is Just One Reason ... by Anonymous Coward · 2010-04-17 15:54 · Score: 2, Insightful
  
  How do you think Windows Device Driver writers do their job?
  Very badly if my experience is anything to go on.
14. Re:This Is Just One Reason ... by cjb110 · 2010-04-17 19:50 · Score: 1
  
  Sorry, but Open != Detailed Documentation any more than Closed does. See Mark Russinovich's blog for way too much detail about the Windows kernel and ecosystem.
  I'd also argue the MSDN site is far more comprehensive and easy to use (if they'd stop pratting about with the colours and layout every other week) than any single source of linux docs (if there even is one)
  MS do seem to realise that you can't write docs assuming the reader knows what the doc is about, unlike most OSS documentation that assumes way to much knowledge.
  These IBM pages are very useful and a commendable activity and a good read for any one interested in *nix, or operating systems. But these are the exception rather than the general rule.
  
  --
  ----- I refuse to have an argument with an unarmed person
15. Re:This Is Just One Reason ... by Anonymous Coward · 2010-04-18 03:02 · Score: 0
  
  Have you ever looked at Amazon or InformIT/Safari or any technical documentation vendor or website?
  Documentation from the vendor or project shows that they care about the details. The FreeBSD project has many web pages and books devoted to it, but it also has awesome documentation at FreeBSD.org. All the man pages are generally up-to-date and shipped with the system. The developers care not only for getting technical details right, they are care about making the user's life easier.
  While there are doc projects for Linux and GNU, they're second-class citizens. Much of the GNU stuff has half-hearted man pages with an info(1) page that no one uses. Does anyone even though how to navigate via the info(1) program?
  
  It absolutely does count, if you know how to read code. You can read documentation all day long, but if it doesn't match the code you'll be lost.
  Code and docs should match. If they don't then it's a bug that the developer should fix. OpenBSD, to take one example, has policy that you have to commit code and documentation changes at the same time. If you have to check the source you've 'lost'. FreeBSD and NetBSD are similar, as is (Open)Solaris. For a lot of software in Linux distros, only if you're lucky.
16. Re:This Is Just One Reason ... by mswhippingboy · 2010-04-18 04:13 · Score: 1
  
  Documentation from the vendor or project shows that they care about the details.
  No, it shows that if they want to sell their product (or in the case of OSS, achieve wide acceptance), they'd better have decent docs. Corporations don't care at all. If not having decent documentation did not impact the bottom line, do you really think these corporations would expend the significant resources they do to produce the documentation?
  In any case, many times the best information available (with the possible exception of reference API docs) comes from third party sources such as books and articles.
  
  For a lot of software in Linux distros, only if you're lucky.
  I agree completely. The decision regarding whether to use a particular OSS application/component has to include many factors (i.e. how well it's supported, how well it's documented, how stable it is, and many more). However, even for poorly documented software, it may still be more effective to use it rather than purchasing it or writing it yourself.
  
  --
  Sometimes the light at the end of the tunnel is the headlight of an oncoming train.
17. Re:This Is Just One Reason ... by Anonymous Coward · 2010-04-18 09:31 · Score: 0
  
  Very badly if my experience is anything to go on.
  So let me guess. You have had bad experiences with 1) video drivers, and 2) printer drivers.
  How about the hundreds of other drivers? How about the ones written by Microsoft? And what is more funny, Linux doesn't even have comparable printer or video drivers!
18. Re:This Is Just One Reason ... by x2A · 2010-04-18 18:36 · Score: 1
  
  "it shows that if they want to sell their product [snip] Corporations don't care at all"
  Err, I think that's actually just breaking down into a language argument now, based on the fuzziness of what the word 'care' is perceived to mean. Ya know, do you 'care' about how much money you have in the bank or do you just 'care' about how much you can do of what you want to do with how much you have? Is the idea that something is just a means to an end sufficient to say that one doesn't truely care about the means because it is, in fact, the ends that is 'cared' about? But in this world, how many true 'ends' are there, and how many of them are just 'means' at a different level of abstraction?
  Off the point here I know, just felt like throwing it in :-)
  
  --
  The revolution will not be televised... but it will have a page on Wikipedia
19. Re:This Is Just One Reason ... by Anonymous Coward · 2010-04-19 20:40 · Score: 0
  
  Shit, I suppose I'd better stop using my printers and video card(s) then, as they clearly don't have drivers, and therefore cannot be working.
20. Re:This Is Just One Reason ... by Anonymous Coward · 2010-04-20 16:42 · Score: 0
  
  With a tought about stuff like:
  http://gizmodo.com/373076/nvidia-responsible-for-nearly-30-of-vista-crashes-in-2007
  I say the previous post has to low score...
IBM by mehemiah · 2010-04-17 09:09 · Score: 1

sense when did IBM care so much about Linux?
1. Re:IBM by Midnight+Thunder · 2010-04-17 09:17 · Score: 1
  
  sense when did IBM care so much about Linux?
  The way I see it, the core businesses for IBM are hardware and services. Anything that helps feed the two is a good investment for IBM.
  
  --
  Jumpstart the tartan drive.
2. Re:IBM by BitZtream · 2010-04-17 09:33 · Score: 0, Troll
  
  Have you been living in a box?
  Linux is to IBM what 'FreeCreditReport.com' is to Experian.
  They've been using it for years to hook people into their hardware so they can then switch them up to proprietary software to actually get the job done. They pretend to be wildly open source and all about doing it cheap ... and once you get roped in you find out to actually do what you want, you gotta pay for it.
  Unless you were born sometime in this year (like the last 4 months) I find it hard to believe you are unaware of IBMs Linux advocacy ... They seem to do more for Linux than Redhat does, Canonical is the only company I can think off thats more visibly Linux centric than IBM.
  
  --
  Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
Re:Linux is a joke by K.+S.+Kyosuke · 2010-04-17 09:11 · Score: 1

That's so lame. It's 2010, why use Windows 2008?

--
Ezekiel 23:20
Re:i love these -- but KernelTrap is back! by Anonymous Coward · 2010-04-17 10:08 · Score: 1, Interesting

There are signs of life at KernelTrap (http://kerneltrap.org/).
There have been a number of postings by Jeremy since the beginning of April.
Re:Linux is a joke by icebraining · 2010-04-17 11:15 · Score: 2, Funny

I start daemons myself, you insensitive clod!

--
Dilbert RSS feed
How does virtualization help by hey · 2010-04-17 12:04 · Score: 1

If you are running 10 processes on 10 servers on one physical machine... isn't easier and more efficient to run 100 processes on one instance of Linux?
1. Re:How does virtualization help by Chang · 2010-04-17 12:53 · Score: 1
  
  In most cases where VM is useful the people who care about the 10 processes bring so much baggage in terms of demands that it pays big dividends to have the overhead of 10 machine images running in order to not have to listen to 10 people whining.
  There's IT theory and then there IT reality...
2. Re:How does virtualization help by Anonymous Coward · 2010-04-17 12:55 · Score: 0
  
  There's also the need to support applications that run on different operating systems.
3. Re:How does virtualization help by Chris+Mattern · 2010-04-17 13:06 · Score: 1
  
  Easier? Yes. More Efficient? Yes. More secure from threats and bugs? Most likely not. 10 processes on 10 virtual servers means that if one process takes out the server, it takes out 9 other processes, not 99 other process, unless it can actually manage to screw over the hypervisor, which is very well protected.
4. Re:How does virtualization help by Trepidity · 2010-04-17 14:09 · Score: 1
  
  Yes, which is what operating-system-level virtualization, which is basically an extension of the old concept of chroots or jails, is intended to do: give you many of the benefits of virtualization without the overhead of having multiple full copies of the OS running. It can also manage some resources better, e.g. having a unified filesystem cache. OpenVZ is Linux's approach.
  However, full virtualization, like Xen, is somewhat more rock-solid in its separation of the virtual machines, and also allows more flexibility, in that each of the virtualized OSs can be different (don't even have to all be Linux, or if they are, can run different kernels/modules).
  
  --
  10 PRINT CHR$(205.5+RND(1)); : GOTO 10
5. Re:How does virtualization help by IkeTo · 2010-04-17 15:39 · Score: 1
  
  In my case, it is because the people running those 10 servers want to have their freedom to set different kernel parameters, to install different OS packages, to run their own Apache server with their own hostname all looking at port 80, and to hold root account without fighting each other. Most importantly, top efficiency is not a concern as the servers are not heavily loaded.
6. Re:How does virtualization help by drsmithy · 2010-04-17 17:53 · Score: 1
  
  If you are running 10 processes on 10 servers on one physical machine... isn't easier and more efficient to run 100 processes on one instance of Linux?
  That depends entirely on how you measure "easier" and "efficient".
7. Re:How does virtualization help by murdocj · 2010-04-18 00:51 · Score: 1
  
  How does it work for multiple copies of Apache to all be looking at port 80? I mean, from the outside world, there can only be one port 80 at that IP address, right?
8. Re:How does virtualization help by chris+mazuc · 2010-04-18 02:04 · Score: 1
  
  Yes
  
  --
  E pluribus unum
9. Re:How does virtualization help by murdocj · 2010-04-18 05:40 · Score: 1
  
  That article starts with "These scenarios are those involving multiple web sites running on a single server, via name-based or IP-based virtual hosts"
  It sounds like this is talking about how to configure a single instance of Apache to serve up different websites based on the incoming IP address or the web site domain name. It doesn't sound like it applies to running multiple virtual machines, each of which has its own copy of Apache, each of which is trying to listen to port 80.
  Although if I'm wrong, I'm sure someone will correct me.
10. Re:How does virtualization help by Mad+Merlin · 2010-04-18 07:45 · Score: 1
  
  How does it work for multiple copies of Apache to all be looking at port 80? I mean, from the outside world, there can only be one port 80 at that IP address, right?
  Each VM normally gets its own IP, distinct from all other VMs and the host.
  
  --
  Game! - Where the stick is mightier than the sword!
11. Re:How does virtualization help by x2A · 2010-04-18 18:47 · Score: 1
  
  The only difference is on network namespace seperation. By default many things will listen on "port x" for IP address 0.0.0.0, which means, for example, Apache may eat up all port 80s for all IP addresses on the machine. Virtual machines get their own IP address and so you don't have to configure Apache to only listen on one, for other instances of Apache to be able to listen on their own. Somebody screwing up their configuration and accidentally listening on all port 80s won't stop the next persons Apache instance from starting up.
  There are more efficient (in processing terms) ways of achieving this; Linux has network namespace support in cgroups for example. Or, you could just configure the software services correctly to listen on the correct IP address! But hey, our computers are already too fast, and we're all too busy, chucking on virtual machines solves both problems!
  (of course there are other benefits too, like live migration of entire machines from hardware to hardware, something vm solutions provide where underlying OS's don't (of course there may be exceptions, but this goes beyond my point))
  
  --
  The revolution will not be televised... but it will have a page on Wikipedia
12. Re:How does virtualization help by chris+mazuc · 2010-04-19 02:49 · Score: 1
  
  I was replying to your comment, not the article:
  
  How does it work for multiple copies of Apache to all be looking at port 80? I mean, from the outside world, there can only be one port 80 at that IP address, right?
  Realistically you wouldn't have completely separate instances of Apache on the same machine, hence the virtualhosts stuff. When you said multiple copies of Apache I assumed you meant they would be on the same server because if they were on VMs it doesn't make sense to say "multiple copies of Apache trying to listen to port 80".
  
  It doesn't sound like it applies to running multiple virtual machines, each of which has its own copy of Apache, each of which is trying to listen to port 80.
  Your VMs would have separate IP addresses with one copy of Apache per VM. If that is a problem then make it NAT and put a proxy in front of it, though I've only seen it done for load balancing with physical servers.
  
  --
  E pluribus unum
Doesn't necessarily do anything by phantomcircuit · 2010-04-17 12:59 · Score: 1

Kernel shared memory only acts on memory pages which it has been advised could be duplicates. This requires applications to specifically tag pages of memory as possibly being duplicates.
Useful for virtualization (which is the primary purpose), but probably not actually functionally useful for more general memory de-duplication.
1. Re:Doesn't necessarily do anything by mortonda · 2010-04-17 13:44 · Score: 1
  
  Why don't you try reading the article.
  "What you'll soon discover is that although memory sharing in Linux is advantageous in virtualized environments (KSM was originally designed for use with the Kernel-based Virtual Machine [KVM]), it's also useful in non-virtualized environments. In fact, KSM was found to be beneficial even in embedded Linux systems, indicating the flexibility of the approach."
2. Re:Doesn't necessarily do anything by phantomcircuit · 2010-04-17 13:49 · Score: 1
  
  Maybe you should take your own advice.
  
  KSM relies on a higher-level application to provide instruction on which memory regions are candidates for merging. Although it would be possible for KSM to simply scan all anonymous pages in the system, it would be wasteful of CPU and memory (given the space necessary to manage the page-merging process). Therefore, applications can register the virtual areas that are likely to contain duplicate pages.
3. Re:Doesn't necessarily do anything by Anonymous Coward · 2010-04-17 13:50 · Score: 0
  
  .. which tells you a bit about the short-sightedness of the embedded Linux systems developers.
  My (FreeBSD!) embedded system makes heavy use of C and shared libraries. I squeeze a lot out of the system specifically because I read a whole lot of random stuff about UNIX in the 1970's/1980's and the various memory footprint issues they had. To the point where I've taken some base system code (eg, tftp and tftpd) and turned their shared code into a shared library. Bam! Instant savings.
4. Re:Doesn't necessarily do anything by mortonda · 2010-04-18 01:41 · Score: 1
  
  But it can be applicable to non-virtualized apps. Furthermore, I see no reason that couldn't be "advisory" in nature, but still do a global scan if needed. The article didn't really say anything about that possibility.
5. Re:Doesn't necessarily do anything by phantomcircuit · 2010-04-18 10:14 · Score: 1
  
  Are you serious or are you a troll? The reason not to do a global scan is in the quote I just gave you form the article.
  
  Although it would be possible for KSM to simply scan all anonymous pages in the system, it would be wasteful of CPU and memory (given the space necessary to manage the page-merging process).
  
  You don't do a full memory scan because the red-black tree would more than double the memory usage during the scan.
Bad Bad Idea by Anonymous Coward · 2010-04-17 14:15 · Score: 0

This could be one of the worst ideas ever ... thats all i can say ...
if you need more info ask everyone else !
A word of caution ! by Taco+Cowboy · 2010-04-17 15:13 · Score: 1

The use of KSM isn't without risk:
"A region can be removed from the mergeable state through the MADV_UNMERGEABLE parameter (which immediately un-merges any merged pages from that region). Note that removing a region of pages through madvise can result in an EAGAIN error, as the operation could run out of memory during the un-merge process, potentially leading to even greater trouble (out-of-memory conditions). "
KSM *will* overcommit the memory as it is designed to do so. And please also be mindful that Murphy's Law always tends to hit when you least suspect it.

--
Muchas Gracias, Señor Edward Snowden !
1. Re:A word of caution ! by Anonymous Coward · 2010-04-18 09:47 · Score: 0
  
  And please also be mindful that Murphy's Law always tends to hit when you least suspect it.
  This sounds like a recursive law ...
Good on them. by Anonymous Coward · 2010-04-17 16:38 · Score: 1, Insightful

First off, as several people have said, IBM did VM over 40 years ago, hypervisors, full hardware virtualization, virtual memory, loads of failover type stuff since they are mainframes after all. Right now the modern wave of VMs are about where IBM was back then (with perhaps failover still being perfected.) IBM mainframe CPUs run the pipeline 2x, with a comparator in between the 2 copies to make sure everything matches. After 1 fault, it backs the pipeline up one and reruns it (and I'm sure logs a fault)... second failure, the CPU is shut off and load put on a spare.
Anyway, for sure memory sharing will massively cut RAM usage in some cases... I'm thinking ISPs where they rent you out a VM, many many people will keep stock (i.e. ISP-supplied) linux kernel + mostly stock apps executing.. which is actually fairly small but could easily save at least 100s of MBs with a good number of VMs on a box.
KSM by Concerned+Onlooker · 2010-04-17 17:14 · Score: 1

"KSM allows the hypervisor to increase the number of concurrent virtual machines by consolidating identical memory pages."
But first you have to waterboard it.

--
http://www.rootstrikers.org/
flying gristle by Mana+Mana · 2010-04-17 21:00 · Score: 0, Offtopic

.
1. Re:flying gristle by PenisLands · 2010-04-18 01:22 · Score: 0
  
  BIG PENIS.
I've tried it and I was disappointed. by Deleriux · 2010-04-17 23:31 · Score: 1

KSM is a great idea, much of its abilities are available in Fedora 12. I tried it and I had higher expectations to be honest.
That is not to say that it is no good - its great but there is a bit of a cost analsysis that should be done before implementing it. You dont get something for nothing - and in this case ultimately your offloading the higher memory usage onto the CPU. Depending on your hypervisor setup this might not be such a bad thing of course.
In my somewhat narrow testing of it I found that:-
a) Even with the same O/S images running multiple times the memory I saved was about 5-10%.
b) It effectively used about 50% of one CPU running the feature.
I think that to really see a benefit to this you have to be running a huge hypervisor with a ton of memory and cpus and a lot of guests as there is a plateau which beforehand makes it quite inefficient to use the features seeing as (at least with my results) the payback is less than 10% anyway.
Old idea is old. by queazocotal · 2010-04-18 01:15 · Score: 1

Linux has had this long ago.
http://www.complang.tuwien.ac.at/ulrich/mergemem/ - for example.
Note that the savings referred to are on kernel 2.0.33.
I used it on my 8M laptop - worked well.
Politics by Colin+Smith · 2010-04-18 08:48 · Score: 1

No, really.

--
Deleted