Hardware Virtualization Slower Than Software?

Jim Buzbee writes "Those you keeping up with the latest virtualization techniques being offered by both Intel and AMD will be interested in a new white paper by VMWare that comes to the surprising conclusion that hardware-assisted x86 virtualization oftentimes fails to outperform software-assisted virtualization. My reading of the paper says that this counterintuitive result is often due to the fact that hardware-assisted virtualization relies on expensive traps to catch privileged instructions while software-assisted virtualization uses inexpensive software substitutions. One example given is compilation of a Linux kernel under a virtualized Linux OS. Native wall-clock time: 265 seconds. Software-assisted virtualization: 393 seconds. Hardware-assisted virtualization: 484 seconds. Ouch. It sounds to me like a hybrid approach may be the best answer to the virtualization problem. "

Comments

Not just the CPU

[kripkenstein]

What matters is that small changes in the hardware make it possible to stop having to depend on costly, proprietary, and complex software--like that sold by VMware.

I am 100% in favor of cheap and open solutions. But I don't agree that this will soon be the case for virtualization. VMWare and the few other major vendors do a lot more than software virtualization of a CPU (which is all TFA was talking about). To have a complete virtualization solution, you need to also virtualize the rest of the hardware: storage, graphics, input/output, etc. In particular graphics is a serious issue (attaining hardware acceleration in a virtual environment safely), which from last I heard VMWare were working hard on.

Furthermore, Virtualization complements well with software that can migrate VMs (based on load or failure), and so forth. So, even if hardware CPU virtualization is to be desired - I agree with you on that - that won't suddenly make virtualization as a whole a simple task.

Re:Look to IBM

[pe1chl]

IBM's VM also started as a software product that had to cope with virtualisation problems in the hardware.
Just like what is happening now, they added specific support to the hardware to make VM perform better.
This all happened before the development of today's architectures, but in the early days of microcomputing, IBM had the position that Microsoft has today: they were the big company that had 90% of the market, and in the eyes of the newcomers all they did was by definition the wrong thing. So nobody would bother to look at 360 mainframes, VM and how it was done before designing their own processor.
(this would be similar to telling a Linux geek to look at how certain problems are solved in Windows... it is Windows, it is Microsoft, so it has to be the wrong solution)

Re:The correct conclusion is more limited

[TheRaven64]

The easiest architecture to virtualise is the Alpha. It had a single privileged instruction, and all that did was shift to a higher privilege mode (which had a few shadow registers available) and then jump to an address in firmware. The firmware could be replaced by using one of these calls. If you wanted to virtualise it then you could do so trivially be replacing the firmware with something that would check or permute the arguments and then vector off into the original firmware.

It also had a few other advantages. Since you were adding virtual instructions, they all completed atomically (you can't pre-empt a process in the middle of an instruction). This meant you could put things like thread locking instructions in the PALCode and not require any intervention from the OS to run them. The VMS PALCode, for example, had a series of instructions for appending numbers to queues. These could be used to implement very fast message passing between threads (process some data, store it somewhere, then atomically write the address to the end of a queue) with no need to perform a system call (which meant no saving and loading of the CPU state, just jumping cheaply into a mode that could access a few more registers).

Re:No not really

[Sycraft-fu]

I haven't read the results, and I doubt I have the technical knowledge to properly analyze them properly. However if I were to guess as to why this might be the case I'd say it's because they didn't do it right. This is a new and fairly complex technology, I somehow doubt it's easy to get right on the first try.

I am not willing, based on a single datapoint, to make any conclusions. That's tanget to my point anyhow, my point was that doing something in hardware and software are quite different.

Yes, AMD Pacifica seems to be far better

[Morgaine]

Is AMD's Pacifica virtualisation system any better?

Apparently, yes, and by a good margin.

There are several documents and articles out there which point out VT's problems and how Pacifica is quite dramatically better. Here's an excerpt from "AMD Pacifica turns the nested tables", part 3 of an informative series of articles:

• The basic architecture of the K8 gives AMD more toys to play with, the memory controller and directly connected devices. AMD can virtualise both of these items directly while Intel has to do so indirectly if it can do so at all.
  
  This should allow an otherwise identical VMM to do more things in hardware and have lower overhead than VT. AMD appears to have used the added capability wisely, giving them a faster and as far as memory goes, more secure virtualisation platform."

So, it looks like AMD are ahead on hardware virtualization at the moment.

If I read it correctly, this is because Intel's VT actually requires a lot of software intervention, so it's not actually a very strong hardware solution at all.

I designed h/w virtualization

[Anonymous Coward]

I designed one of the x86 h/w virtualization offerings. It's obvious that outside of device emulation, the biggest overhead of virtualization is the s/w emulation of what amounts to two levels of address translation (especially hairy in multiprocesor systems due to the brain-dead x86 page table semantics that do not require explicit invalidation). So clearly you want nested-paging support in h/w. However, that support is a little more complex than a few microcode changes to trap selected privileged instructions --- and due to schedule pressures, it didn't make it into the current release. Once that's in, expect h/w virtualization to speed up significantly.

Note that this doesn't make all the other stuff in VT/SVM useless; there are lots of places on the x86 where pure s/w virtualization has to go to great lengths of complexity just to get things correct. As a simple example: there's no way on "old" x86 h/w to save & restore segment descriptors (which you need to do on world switch) --- all you get is the selector, and if the guest O/S has overwritten the in-memory copy, you're out of luck. "Fixable" in s/w (obviously; VMWare does it), but just plain grody. So a major advantage of SVM/VT is that it becomes a lot *easier* to write a VMM (opening up the market to more players; this is starting to show in the Macintosh market) --- eventually, it should become faster, too.

On a separate note, over the next years, expect h/w assistance for dealing the device emulation (and not just from the CPU vendors).