Slashdot Mirror

← Back to Stories (view on slashdot.org)

Red Hat & AMD Demo Live VM Migration Across CPU Vendors

Posted by kdawson on from the dude-where's-my-virtualization-business dept.

An anonymous reader notes an Inquirer story reporting on something of a breakthrough in virtual machine management — a demonstration (not yet a product) of migrating a running virtual machine across CPUs from different vendors (video here). "Red Hat and AMD have just done the so called impossible, and demonstrated VM live migration across CPU architectures. Not only that, they have demonstrated it across CPU vendors, potentially commoditizing server processors. This is quite a feat. Only a few months ago during VMworld, Intel and VMware claimed that this was impossible. Judging by an initial response, VMware is quite irked by this KVM accomplishment and they are pointing to stability concerns. This sound like scaremongering to me ... All the interesting controversy aside, cross-vendor migration is [obviously] a good thing for customers because it avoids platform lock-in."

34 of 134 comments (clear)

  1. Bravo! by Cornwallis · · Score: 4, Funny

    I love to see things like this that give me a greater freedom to migrate off the major players.

    1. Re:Bravo! by 2names · · Score: 3, Informative

      We have certainly come a long way when a Cornwallis supports freedom of the people. :)

      --
      "I'm just here to regulate funkiness."
  2. This is still unreleased test demo's by Beached · · Score: 4, Insightful

    The real beauty of this will come when the system automatically moves VMs to machines in case of hardware problems or when a system is underutilized. It would let you power down servers during non-peak times and save oodles of cash.

    --
    ---- aut viam inveniam aut faciam
    1. Re:This is still unreleased test demo's by Hercynium · · Score: 3, Insightful

      Well, that kinda *is* the purpose of live VM migration... it's already being done, just not between systems with different processor types.

      --
      I'm done with sigs. Sigs are lame.
    2. Re:This is still unreleased test demo's by voidptr · · Score: 3, Interesting

      This is like blowing the engine in a Ford and electing to put a Chevy engine in to replace it.

      While still driving down the highway at 60 mph.

      --
      This .sig for unofficial government use only. Official use subject to $500 fine.
    3. Re:This is still unreleased test demo's by Comatose51 · · Score: 4, Interesting

      You mean like VMware's VMotion, HA, and DRS functionalities?

      --
      EvilCON - Made Famous by /.
    4. Re:This is still unreleased test demo's by TheRaven64 · · Score: 4, Insightful

      They don't seem to have released many details of this. Migrating between x86-with-SSE and x86-without-SSE, for example, is pretty simple - you just need the OS or hypervisor to trap the illegal instruction exception and emulate. Migrating from x86 to x86-64 is pretty easy too - you just don't get any advantages from the 64-bit chip. Going the other way is really hard, and would need the hypervisor to trap the enter-64-bit-mode instruction and emulate everything until the mode was exited (difficult, slow, and probably pointless).

      I read TFA when it first came out and couldn't work out exactly what they were claiming was novel. Migrating between very-slightly-different flavours of x86 is not really that hard. Migrating between ARM and x86 would be incredibly hard - Xen can actually do this with the P2E work (not sure if it ever made it in to trunk), which migrated a VM from real hardware in to QEMU but, again, that's not an ideal solution unless the emulator has traps that userspace can use - for example a Java VM might get a signal after migration, flush its code caches, and re-JIT as x86 code instead of ARM.

      --
      I am TheRaven on Soylent News
    5. Re:This is still unreleased test demo's by JEB_eWEEK · · Score: 3, Insightful

      Yes, except without requiring identical hardware.

    6. Re:This is still unreleased test demo's by nabsltd · · Score: 3, Informative

      VMware doesn't require "identical" hardware to do live migration, either.

      It does have to be similar enough, which at this point pretty much means just the same processor manufacturer. As long as the processor supports the hardware virtualization, then VMware will allow you to set up a cluster that will allow live migration with no issues.

    7. Re:This is still unreleased test demo's by sirsnork · · Score: 2, Interesting

      Between different vendors is actually quite hard. Live migration requires saving the CPU state exactly, including all registers. Going to a different vendors CPU means all this saved state may not match up and then you have to do something so the VM won't just crash. This is actually becoming _harder_ as more and more virtualization technology is being put into the CPU silicon (Intel VT, AMD-V etc). Each new series has a few more features to make virtualization simpler, and you have to deal with making sure what was available to the VM on one CPU is identical to whats available on the new CPU without destroying performance (which is what will happen if you start emulating).

      In saying that, VMWare are very very VERY careful with the tech they introduce, to give you an example round robin network teaming is still "experimental". I'm fairly sure they have played with this internally already and not done it either because it would make support harder or because of the changing CPU landscape with regard to the integrated virtualization features on new CPU's they would need to release a new version for each new CPU release for this to continue working.

      Make no mistake, this is big news for KVM and well done to them, but if they can make it work reliably so can anyone else, and that includes VMWare

      --

      Normal people worry me!
  3. Umm... by frodo+from+middle+ea · · Score: 2, Interesting
    All the interesting controversy aside, cross vendor migration is [obviously] a good thing for customers because it avoids platform lock-in Well almost all VM products barring VirtualPC do indeed supoort running the same VM image on across various vendor platforms, in fact that is the whole point of a VM , isn't it ?

    The fact to highlight is that the migration was done of a live VM without disrupting the VM's operations.

    --
    for the last time people, I am "frodo from middle eaRTH", not "middle eaST".
    1. Re:Umm... by TheRaven64 · · Score: 2, Informative

      On most other setups you'd have to shut the VM down and then restart it on the other machine for it to work correctly

      Do you? I first saw Xen demo live migration in 2005, and I don't think it was new then. Their demo had a Quake server being thrown around a cluster without clients noticing. Downtime was well under 100ms. You can read the paper for more information.

      They were claiming that you can move between processor types, but they didn't specify how much different they could be. If it's just a matter of SSE or 3DNow! support disappearing then that's not a hard problem - just trap-and-emulate any of the old instructions. Relaunching programs that use these will cause the new values of CPUID to be picked up.

      --
      I am TheRaven on Soylent News
    2. Re:Umm... by nabsltd · · Score: 2, Informative

      And, when you think about it, any instruction that you would have to trap if the VM used to be running on a different processor must be trapped at all times.

      This is because you have no way of knowing which processor type the VM was first started on. When this happened, it's likely the OS did some hardware checking and figured out which instructions it could (and could not) use. Moving the VM isn't going to change what the OS believes is the processor, and that's the problem.

      Overall, VMware's Enhanced VMotion Compatibility method of lying to the OS about the capablilities of the processor seems to be the easist way of doing this. But, they only do it within one CPU manufacturer, because otherwise you'd end up with a very low-featured virtual processor.

  4. Xen 3.3 supports this already by stabe · · Score: 3, Informative

    Xen supports this feature since Xen 3.3, it is called CPUID: http://www.nabble.com/Xen-3.3-News:-3.3.0-release-available!-td19106008.html No real breakthrough here...

    1. Re:Xen 3.3 supports this already by stabe · · Score: 2, Informative

      Yes, it does: http://lists.xensource.com/archives/html/xen-devel/2008-06/msg00430.html

  5. Still x86 only by boner · · Score: 3, Insightful

    Real magic would have been demonstrating a move between ANY processor architecture - Power, SPARC, x86_64 etc..

    Between x86 processors is nice, but not unexpected.

    1. Re:Still x86 only by TheRaven64 · · Score: 2, Interesting

      Depends. Modern emulators can run at around 50% of the host platform speed. If your guest is paravirtualised then all of the privileged instructions will be run in the hypervisor. If you're running a JIT in the guest then you can poke it to flush its code caches and start emitting native code for the new architecture, but even if you aren't then migrating the VM from the 200MHz ARM chip in your cell phone to the quad-core 4GHz x86 chip connected to your TV might be interesting.

      --
      I am TheRaven on Soylent News
  6. This was in all likelyhood faked. by Anonymous Coward · · Score: 5, Funny

    Open source is for morons.

    Only Apple has the engineering know-how and skills to pull of something like this. The fact that they have not done so to date is a clear indication that it is impossible.

  7. check the graphs... by alta · · Score: 5, Interesting

    Go to 4:05 in the video. On the far left, you can see from the blue intel line that the guest is running there, then they migrate, and the blue line goes to the idle point, and the orange line starts taking the load. But NOTICE, the AMD line is consistantly higher than the intel line was. I'm no intel fanboy... or AMD. I have both intel and amd servers in my racks. I just thought it was interesting, and I'm surprised they let the video go out like that.

    --
    Do not meddle in the affairs of sysadmins, for they are subtle, and quick to anger.
    1. Re:check the graphs... by michrech · · Score: 2, Informative

      It didn't seem that interesting to me. If you watch the video, the Intel and Barcelona machines showed no VM's running (0% load). When the Shanghai server took over the load, *of course* it's load line will rise -- it's the only server running a VM at that point!

      There are no shenanigans going on here, and I don't think this says anything about the chips as you imply, either.

      --
      bork bork bork!
    2. Re:check the graphs... by wanderingknight · · Score: 2, Interesting

      GPUs have nothing to do with video decoding, it's handled 100% by the CPU. At least until we get a software that can reliably take advantage of the relatively recent introduction of h264 decoding on some high-end GPUs.

  8. Stability issues are justified by mnmn · · Score: 4, Interesting

    Declaration: VMware support engineering here, but speaking strictly on my own behalf.

    The stability issues are justified if you consider all types of VMs. Windows 2003, default RHEL5 kernels etc use more than the basic set of assembler instructions (disk IO code uses MMX, SSE etc).

    We can compile a kernel for strictly 486 CPUs and demonstrate migrations between AMD and Intel using extensive CPU masking: http://kb.vmware.com/kb/1993

    We've also known that mismatched CPU stepping makes the VMs unstable. This is because instructions suddenly run faster or slower compared to the front side bus, not all of Linux and Microsoft code has been tested against that. You can happily try it and a lot of our customers succesfully do. Some get BSODs and kernel oops. This is not our fault.

    If you virtualize the instructions more (bochs?) you can of course move the VM anywhere including a Linksys router's MIPS chip. At the cost of speed of course.

    Lastly, why would we want to keep customers stuck to one CPU vendor? We've software vendors.

    --
    "Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
    1. Re:Stability issues are justified by Anthony+Liguori · · Score: 4, Interesting

      Declaration: VMware support engineering here, but speaking strictly on my own behalf.

      The stability issues are justified if you consider all types of VMs. Windows 2003, default RHEL5 kernels etc use more than the basic set of assembler instructions (disk IO code uses MMX, SSE etc).

      KVM goes to great lengths to by default, mask out CPUID features that aren't supported across common platforms. You have to opt-in to those features since they limit a machine's migrate-ability.

      However, I won't say this is always safe. In reality, you really don't want to live migrate between anything but identical platforms (including identical processor revisions).

      x86 OSes often rely on the TSC for time keeping. If you migrate between different steppings of the same processor even, the TSC calibration that the OS has done is wrong and your time keeping will start to fail. You'll either get really bad drift or potentially see time go backwards (causing a deadlock).

      If you're doing a one time migration, it probably won't matter but if you plan on migrating very rapidly (for load balancing or something), I would take a very conservative approach to platform compatibility.

    2. Re:Stability issues are justified by kscguru · · Score: 5, Informative
      Yet Another VMware engineer here.

      The new Intel/AMD CPU features that allow masking of CPUID bits while running virtualized also make processors recent enough that most of the interesting features are present - MMX, SSE up to ~3. The "common subset" ends up looking like an early Core2 or a Barcelona (minus the VT/SVM feature bits, of course) - Intel and AMD run about a generation behind on adding each other's instructions. Run on anything older than the latest processors, and you have to trap-and-emulate every CPUID instruction. Enough code still uses CPUID as a serializing instruction that this has noticeable overhead.

      So there are two strategies. Pass directly through the CPUID bits (and on the newest processors, apply a mask), or remember a baseline value, trap-and-emulate every CPUID and always return that value. Sounds like KVM has picked the latter approach for a default; VMware's default is to expose the actual processor features and accept a mask as an optional override, which skews towards exposing more features at the expense of some compatibility. Equally valid choices, IMHO.

      The Worst Case Scenario when not doing a trap-and-emulate of every CPUID is an app that does CPUID, reads the vendor string, then decides based on the vendor string which other CPUID leafs to read. (Like the 0x80000000 leafs, which are vendor-specific and would come back as gibberish if you get the processor wrong). If the app migrates during the dozen or so instructions between the first CPUID and the following ones, instant corruption. Good enough for a pretty demo, destined to make a guest kernel die a few times a year if actually used in production. And I'm 95% sure this is what the OP demo is doing - living dangerously by hoping mismatched CPUID results never get noticed.

      I agree with Anthony Liguori here - on a production machine, an Intel/AMD migration is way too much of a stupid risk. All you have to do is reboot the VM, it's much safer.

      (As a side note to everyone reading, the reason Linux timekeeping is such a problem is that TSC issue. Intel long ago stated TSC was NOT supposed to be used as a timesource. Linux kernel folks ignored the warning, made non-virtualizable assumptions, and today are in a world of hurt for timekeeping in a VM. And only now, many years later, are patching the kernel to detect hypervisors to work around the problem.)

      --

      A witty [sig] proves nothing. --Voltaire

    3. Re:Stability issues are justified by Chirs · · Score: 3, Interesting

      The TSC is an optional clock source. You can use other things (ACPI, HPET) but the problem is that they're relatively expensive to access.

      The kernel people have been complaining literally for multiple years that x86 needs a system-wide clocksource that is cheap to access (and presumably hypervisor-friendly). So far AMD and Intel haven't bothered to provide one.

    4. Re:Stability issues are justified by TheLink · · Score: 2, Interesting

      Yes you're not supposed to use TSC.

      BUT there is no good alternative that's:
      1) Cheap
      2) Fast
      3) Available on most platforms

      I find it quite amazing actually that the CPU manufacturers add all those features, and yet after so many years there is still no good standard way to "get time", despite lots of programs needing to do it.

      --
    5. Re:Stability issues are justified by Anthony+Liguori · · Score: 2, Informative

      The new Intel/AMD CPU features that allow masking of CPUID bits while running virtualized also make processors recent enough that most of the interesting features are present - MMX, SSE up to ~3. The "common subset" ends up looking like an early Core2 or a Barcelona (minus the VT/SVM feature bits, of course) - Intel and AMD run about a generation behind on adding each other's instructions. Run on anything older than the latest processors, and you have to trap-and-emulate every CPUID instruction. Enough code still uses CPUID as a serializing instruction that this has noticeable overhead.

      Modern OSes do not use CPUID for serialization. We trap CPUID unconditionally in KVM and have not observed a performance problem because of it. Older OSes did this but I'm not aware of a modern one.

      My understanding of the reason for the recent CPUID "masking" support is because if you are not using VT/SVM (Xen PV or VMware JIT), there is no way to trap CPUID when it's executed from userspace. AMD just happened to have this feature so when Intel announced "FlexMigration", they were able to just document it. I don't think it's really all that useful though.

      (As a side note to everyone reading, the reason Linux timekeeping is such a problem is that TSC issue. Intel long ago stated TSC was NOT supposed to be used as a timesource. Linux kernel folks ignored the warning, made non-virtualizable assumptions, and today are in a world of hurt for timekeeping in a VM. And only now, many years later, are patching the kernel to detect hypervisors to work around the problem.)

      The TSC is often used as a secondary time source, even outside of Linux, but yes, Linux is the major problem. But Windows it not without it's own faults wrt time keeping. Dealing with missed timer ticks for Windows guests is a never ending source of joy. Virtualization isn't the only source of problems here. Certain hardware platforms have had overzealous SMM routines and the results was really bad time drift when running Windows.

  9. Re:Pfff... by Anonymous Coward · · Score: 3, Insightful

    so easy that you did it yourself three years ago, right?

  10. Not quite a break through by Anthony+Liguori · · Score: 2, Insightful

    FWIW, KVM live migration has been capable of this for a long time now.

    KVM actually supported live migration of Windows guest long before Xen did. If you haven't given KVM a try, you should!

  11. Xen does migration, but not Live... by LinuxGeek · · Score: 4, Informative

    This is a demo of a Live migration, no shutdown or reboot involved. Xen does not support the live migration of a running VM between an AMD and Intel server. Watch the video, they are running a video in the VM that keeps playing during the migration. Very impressive stuff.

    --

    Kindness is the language which the deaf can hear and the blind can see. - Mark Twain
  12. Creds anyway by noundi · · Score: 2, Insightful

    It's worth noting that VMware have been a huge contribution to the Linux-society, giving corps a very good reason (â$£) to migrate, thus including important pawns in the future of Linux. I for one believe that VMware was wrong, but that it's an honest mistake. There's no use in poking on VMware for this one, hopefully they'll help lift the technology even higher along with their competitors.

    You've lost this round VMware, but the match isn't over yet!

    --
    I am the lawn!
  13. Re:Wasn't this always possible? by TheRaven64 · · Score: 2, Informative

    Actually, it is suspended, but only for a fraction of a second. First you copy the entire contents of memory to the new machine and mark it as read-only. Each page fault caused by this is used to mark pages that are still dirty. Then you copy these. You keep repeating this process until the set of dirty pages is very small. Then you suspend the VM, copy the dirty pages, and start the VM on the new machine. Userspace programs will just notice that they went an unusually long time without their scheduling quantum. With Xen, at least, the kernel is responsible for bringing up and shutting down all CPUs except the first one, so the kernel will notice the migration (in a paravirtualised kernel - with HVM it won't) and restart the other (virtual) CPUs.

    --
    I am TheRaven on Soylent News
  14. Re:Um by Korin43 · · Score: 4, Insightful

    Except they're doing it with KVM, which is open source..

  15. Re:Um by abdulla · · Score: 3, Insightful

    Why should it be dumbed down? I don't go reading Biology articles and expect to know everything. That's why there are links to other articles explaining each bit in more detail.