Slashdot Mirror

← Back to Stories (view on slashdot.org)

Pointers for Developing x86 Virtualization?

Posted by Cliff on from the how-does-VMWare-do-it? dept.

josh asks: "For my next project, I've decided I want to do something related to x86 virtualization (the way VMware does it or Plex86 not Xen/Bochs/etc.) but I really don't know where to start. Googling hasn't been helpful (just look at the results if you don't believe me). Are there any resources for learning about this kind of x86 virtualization? I know virtual 8086 mode wouldn't work, but without that what advantage does something like VMware have over something like Bochs? Are there any F/OSS projects aimed at something along the lines of my thinking? Please enlighten me with any references and resources you might have. Thanks!"

41 comments

  1. Read More Slowly by mcgroarty · · Score: 5, Informative
    Reread the second paragraph of the Plex86 page you linked to -- plex86 virtualizes like VMWare. It's not an emulator. bochs and plex86 were started by the same developer, and virtualization versus emulation is the difference between the two projects.

    Go to the plex86 page you linked to and download source. That's an excellent place to start. Apart from that, boning up on the difference between user and kernel modes and getting your hands on the free Intel developers' docs on the Intel website are about as good as it gets for this kind of learning.

  2. Re:Read More Slowly [OP] by josh42 · · Score: 1

    plex86 only runs Linux under its VM. I assumed that the implementation for something like this is different from the implementation of a full x86 VM monitor. Is the code similar enough to warrant perusal, or will it just hopelessly confuse me even more? Thanks for your help! -- Josh

  3. qemu by cfallin · · Score: 2, Informative

    I find qemu to be quite excellent x86 virtualization software. It's closer to VMware than Bochs, using dynamic translation.

    1. Re:qemu by caseih · · Score: 3, Informative

      Not quite. While qemu will most likely yet gain virtualization to speed it up, qemu is definitely not what the poster is asking about. qemu is an emulator just like bochs, except that qemu employs dynamic translation of cpu instructions (and caching of said blocks of code). One mode of qemu, qemu-fast, uses a linux kernel module to allow the native OS memory manangement and paging routines. In pure emulation mode memory management is also emulated. At best qemu can yield a raw cpu speed of 1/3 the host processor. Compare this to vmware which, although it seems slow, in theory can be almost full speed in terms of cpu-bound metrics.

      I think that down the road qemu will adopt some virtualization techniques on various platforms. Obviously this would be limited to x86 on x86 or ppc on ppc. But it will be exciting to watch and follow qemu. I already run win2k in qemu on my 1.5 gHz athlon at quite a respectable speed.

    2. Re:qemu by OmniVector · · Score: 1

      one of qemu's goals is to do x86 to ppc code translation for projects like darwine. i'm not sure that your last statement is entirely true.

      --
      - tristan
    3. Re:qemu by jhoger · · Score: 1

      Ok, I'll bite...

      How do you figure that a dynamic translator like QEMU is bounded to 1/3 the speed of the host processor? If you are dynamically translating code, you could theoretically get 100% efficiency after the initial translation hit. And if you load pre-translated code you can close the gap by a lot. Few people would count load time anyway.

      The only place there is a ceiling is on the guest OS dynamically generating code, in which case you have to emulate. But I imagine there are few places in most OS's of note that do dynamic code generation and those could be detected and handled as special cases and you wouldn't have to invalidate the cache.

      Imagine the best dynamic translator in the world is an assembly language programmer. I take a disassembly in from CPU1 and port that assembly code to CPU2. Is my final program on CPU2 bounded to 1/3 the efficiency of the program on
      CPU1?

    4. Re:qemu by ballpoint · · Score: 0

      You've been reading - or worse, writing - too much patents lately. The word "said" is a dead giveaway.

      --
      Flourescent (adj): smelling like ground wheat.
    5. Re:qemu by caseih · · Score: 2, Interesting

      One of the few companies that has years of experience with dynamic recompiling emulation is ardi (www.ardi.com). Their 68k synthetic cpu was worked on for several years, achieving incredible performance, but alas, only 1/3 cpu speed on average. See http://www.ardi.com/SynPaper/node12.html.

      Now of course in theory if you had a lot of cache you could approach native cpu speeds, assuming that you always executed the same code over and over again. Caching certainly is the key to performance here, just like in CPUS. But realistically, you can't always keep every dynamic block in cache. Eventually it will be invalidated and new blocks will have to be translated. Much faster than emulation (orders of magnitude) but still not quite full host cpu speed.

      Your point about translating the assembly code is an interesting point, but fails to account for the fact that this has to happen on the fly. Thus the 1/3 bound is not theoretical, but simply a practical one. I'm sure newer techniques will come along to improve this.

    6. Re:qemu by hattmoward · · Score: 1

      One correction here: qemu-fast doesn't require a kernel module, but it does require a change to a kernel constant. Basically, you have to adjust the kernel so it expects to be loaded to a different location in memory when it starts. Another neat feature of Qemu is that you can point it at a linux kernel file, and it will load it at the right place in (emulated) memory, then begin executing, thus skipping the emulated PC BIOS.

      One thing not to forget is that Qemu wants to be a portable project and emulate a few different platforms. They're just beginning to add optimizations to IA-32/x86 emulation, and they have added a few x86-on-x86 tweaks already.

    7. Re:qemu by Anonymous Coward · · Score: 2, Interesting

      Come on, that's still the old circa 1995 results they published back then -- they are quoting results on a 75MHz 486 for Chris' sake! I haven't seen any evidence that the ARDI guys have done much development in the last ten years, and I suspect they are paying their bills with other full-time jobs.

      There have been plenty of demonstrations of 75% to 100% of native performance being achieved by dynamic recompilations, by products such as the Digital FX86 and Connectix VirtualPC. The HP Dynamo research project even managed to exceed 100% efficiency by clever optimizations during the recompilation! (Of course they didn't start from uber-optimized code, but then again little real-world code has seen whole-program optimization over library or even module boundaries.)

    8. Re:qemu by Anonymous Coward · · Score: 0

      Isn't dynamic recompilation exactly what VMWare is doing? How else are you going to virtualize a non-virtualizable architecture? Read up on the original Plex86 design, before they skimped on running unmodified operating systems, and you'll see that they were essentially creating two thirds of a dynamic recompilation architecture.

      The basic idea is to replace non-virtualizable instructions by function calls. This means that you end up changing code size, and need to remap code blocks to new addresses, and translate all jump instructions, in addition to the non-virtualizable instructions. Not something you would do in a school project lesser than a thesis.

    9. Re:qemu by Anonymous Coward · · Score: 0

      You might take a look at this thesis work. it contains information about the developement of the xok monitor.

    10. Re:qemu by Anonymous Coward · · Score: 0
      Imagine the best dynamic translator in the world is an assembly language programmer. I take a disassembly in from CPU1 and port that assembly code to CPU2. Is my final program on CPU2 bounded to 1/3 the efficiency of the program on CPU1?

      You conveniently ignore the TIME is takes you to perform the translation.

      As Transmeta discovered, running the translate code is only a fraction of the problem. The translation and cache management of the translated fragments is quite significant.

  4. Link by addaon · · Score: 0

    Here's a link that should have more than enough information to get you started.

    --

    I've had this sig for three days.
    1. Re:Link by vinit79 · · Score: 2, Informative

      The acm portal does have a wealth of information but u need to pay for the subscription to access most papers(unless as in my case u r a student and ur univ library has a subscription) . However, one good idea is to find the relevant papers and then google for them( many authors often maintain a copy of their papers on their webpages .. albiet somewhat illegally)

    2. Re:Link by addaon · · Score: 2, Informative

      It is legal for the author to make a copy of their paper available free of charge, although the ACM (and other professional organizations) discourage it. The author is the copyright holder.

      --

      I've had this sig for three days.
  5. Why do you care? by hak1du · · Score: 3, Insightful

    Virtualizing the Pentium is a lot of tedious work because the Pentium just wasn't designed for it. You won't be learning much about virtualization in general, just a lot of Pentium-specific tricks. Why bother with that?

    Sooner or later, Intel is going to make the Pentium virtualizable in a more straightforward way. Until then, I'd just stick with one of the available solutions (including Plex86).

    1. Re:Why do you care? by Gilk180 · · Score: 1

      Intel will never make the pentium virtualizable. That would require that they break backwards compatibility and produce a processor spec that is not a kludge wrapped within a kludge stuck in a 200 stage pipeline.

      I'm not a fanboy for any processor. I don't build 'em or write assembly most of the time, but the x86 architecture is a mess.

    2. Re:Why do you care? by GoRK · · Score: 2, Interesting

      Of course, they also have the option to make something else other than X86 the base architecture upon which they support virtualized X86 (and maybe if we're lucky other cores also). It would make a lot more sense to do this kind of thing on top of something new than to stuff it all in some kind of messy architecture extension as you suggested.

      Of course, this still adds to the mess either way, but it does make some sense.

      Maybe that's why Transmeta did it.

    3. Re:Why do you care? by TheLink · · Score: 1

      AMD had an opportunity to add easy virtualization at least in some modes when they introduced the AMD64 instructions. But unfortunately I don't think they did, did they?

      --
    4. Re:Why do you care? by hak1du · · Score: 1

      Intel will never make the pentium virtualizable. That would require that they break backwards compatibility and produce a processor spec that is not a kludge wrapped within a kludge stuck in a 200 stage pipeline.

      I don't see why it requires breaking backwards compatibility. It may be a lot of work, given what a mess the Pentium is, but Intel can do it.

      I suspect they don't want to--better to sell more real processors--but eventually, they won't have a choice.

    5. Re:Why do you care? by T-Ranger · · Score: 2, Interesting
      Im neither a EE, nor do I have paticularly good "low level" programming tasks. That said, my Uni ASM course was on a IBM mainframe, and I use VMWare regularly, so I know a bit of the practical side of things.

      VMWare, in the workstation and GSX version, runs on top of a host OS. ESX is its own "OS". In the IBM world, the VM "OS" layer is very specialized and so far as I know, used for just allocating resources to VM instances... The hardware helps a lot in this case as well.

      Idea: What would be the possibility of a "Meta" PC hardware/OS platform? Possibly having zero reselblence to PCs of today, no PC compatable BIOS, N/S bridge concept gone, etc... The hardware, and a thin VM managemnt OS works together to virtualize the hardware, making it appear to be a PC to the VMed OSs.

    6. Re:Why do you care? by Anonymous Coward · · Score: 0

      Unfortunately, IBM patents on the obvious ways to do virtualisation are still valid. Yet another example of patents harming innovation. The purpose of patents is to allow innovation to be slowed and controlled by old men in smoky rooms, not to reward the inventor...

    7. Re:Why do you care? by Anonymous Coward · · Score: 0

      That's surprising to me. I thought patents would expire after twenty years, and IBM has doing virtualisation for longer than that.

  6. What's wrong with what you've got? by Christopher+Anthony · · Score: 1
    Which way do you not want to do it? Given that Xen and Plex86 both run Linux, they are good enough. So it's not a completely plain vanilla x86 Linux, but once AMD gets support in and Linux gets ported to that, it will be. AMD is working on some improvements to AMD64 for that. x86 isn't very good for virtualization because the hardware isn't designed for it. It's too difficult to partition the resources.

    PS. They usually work by running user level code as is, and reflecting system calls to ring 1 where the guest OS runs. Or they just interpret or block translate all kernel code like VMWare does to be completely x86 compatible, even for crappy systems like Windows that need more than 2 hardware privelige levels and aren't portable. Either way, they have to call into the real Linux kernel when the guest OS actually needs access to hardware like video cards and the virtual memory subsystem.

    --

    ---
    Find out more about the impending downfall o

    1. Re:What's wrong with what you've got? by Anonymous Coward · · Score: 1

      Your blog is great -- from Windows Virgin to Visual J++ Developer in two weeks. Micro$uck ueber alles!

  7. Get guides to the hardware you want to emulate by ChaseTec · · Score: 4, Informative

    Half of what you want is cpu documentation, Intel makes nice free guides to their 32 bit x86 cpus. Sometimes they even will send out the manuals free in book format, right now it looks like you can download them or order a free cd. Check back often because the free books(shipping free too) come and go. http://developer.intel.com/design/pentium4/manuals /index2.htm

    The other main half is BIOS info, check our Ralf Brown's int list: http://www-2.cs.cmu.edu/afs/cs.cmu.edu/user/ralf/p ub/WWW/files.html

    For the rest you just need lots docs about various hardware like floppy controllers and such, check out my site for OS developers since I probably have more links listed then anyone else. Besides this stuff you might want to re-include java in your google searches because most of the research into virtual machines that I've seen uses java but alot would still apply.

    --
    My Hello World is 512 bytes. But it's also a valid Fat12 boot sector, Fat12 file reader, and Pmode routine.
  8. MOL by Joe+Tennies · · Score: 3, Informative

    Something you may want to look at is MOL (maconlinux.org). It's for PPC, but it does demonstrate how to do virtual machines like you are discussing.

    Don't forget that one good thing is also your ability to write a good driver for the client OS. That can yield huge speed increases on its own.

  9. Answer by runderwo · · Score: 4, Funny
    Pointers for Developing x86 Virtualization?
    Yes, I would strongly recommend their use.

    --
    LRC, the best-read libertarian site on the web
  10. virtualization wiki by johnseq · · Score: 2, Informative
    I maintain a wiki on virtualization projects which has several oss listed

    this page has all the open source wiki pages, and this is the front page for the site.

  11. Wait 'till he find out... by leonbrooks · · Score: 1
    ...why people really fear Internet Exploder. Hint: it's not the missing tabs.

    BTW, Con Zymaris listed out (on the OSIA list) some of the bet-your-company-on-this technologies which Microsoft has abandoned (who remembers Blackbird? How about DNA, the Distributed Network Architecture?), and more systems which Microsoft has sucked people into and then changed, requiring significant code rewrites (e.g. VB4 and again at VB7). It's a long and impressive list.

    Bearing this in mind, and that Visual J++ is already way past "dead meat" stage, why on earth would anyone in their left mind be developing any new stuff in it? Christopher, if you're real, don't rely too much on your new day job...

    --
    Got time? Spend some of it coding or testing
    1. Re:Wait 'till he find out... by tcopeland · · Score: 1

      > Visual J++ is already way past "dead meat" stage

      It's a nice fast Java IDE, though. As long as you stay away from the RNI/JDirect "lots of comments that get magically transformed into native method invocations" stuff, it's pretty cool.

      --
      The Army reading list
    2. Re:Wait 'till he find out... by Anonymous Coward · · Score: 0

      > How about DNA, the Distributed Network Architecture?)

      Now called DotNet Enterprise Services.

      DNA was nothing more than a poorly used marketing term for COM+.

  12. Not restricting to x86... by hubertf · · Score: 3, Informative

    here are a few links from my collection on the topic:

    + vmware
    + bochs
    + vax with simh-vax, see
    http://www.netbsd.org/Ports/vax/emulator-howto.htm l
    + xen
    (http://www.cl.cam.ac.uk/Research/SRG/netos/xen/)
    + LilyVM
    (http://lilyvm.sourceforge.net/index.ja.html)
    + mips64emul
    (http://www.mdstud.chalmers.se/~md1gavan/mips64emu l/),
    + dosbox (http://dosbox.sf.net)

    I'm mostly interested with running non-Linux (e.g. NetBSD, Solaris) in a virtual environment for using it in my "Virtual Unix Lab" training environment, see http://www.feyrer.de/vulab/).

    - Hubert

  13. Suggestion by curator_thew · · Score: 2, Interesting


    Fantastic idea, I would suggest you look at the L4Ka and related projects (http://l4ka.org/projects/hazelnut/).

    Please do not create a new project from scratch: there are enough.

    What is needed though is adapting existing operating systems (e.g. Linux, BSD, etc) to work in these virtualised environments. Further to this, also supporting co-ooperative virtualise environments.

    Your efforts in this area would be very well looked upon.

  14. Newsflash! Windows NT not portable! by supergumby · · Score: 1

    Accordining to noted troll Christopher Anthony, Windows NT is not portable! Administrators still running NT on Alpha are understandably shocked.

    Windows NT was designed with a hardware abstraction layer to seperate machine dependent code from the rest of the code base. Windows NT 4.0 was launched on x86, Alpha, MIPS, and PowerPC. So much for not portable.

  15. Re:Read More Slowly [OP] by Anonymous Coward · · Score: 1, Insightful
    plex86 only runs Linux under its VM. I assumed that the implementation for something like this is different from the implementation of a full x86 VM monitor. Is the code similar enough to warrant perusal, or will it just hopelessly confuse me even more? Thanks for your help! -- Josh

    Plex86 is essentially a full x86 virtualization, with the exception of a few difficult corner cases that were easier to avoid (by patching the guest Linux kernel) than emulate. The same modifications could be made for any Free Software operating system, so the only reason for full virtualization would be to run proprietary OSes. The architecture of Plex86 is similar to that of proprietary x86 virtualizers: a kernel module and a user-mode program working together.
  16. Introduction to Virtualization by gtrubetskoy · · Score: 1

    Here is a nice link on the subject: http://www.kernelthread.com/publications/virtualiz ation/

  17. Check out Qemu - the Quick Emulator by Nailer · · Score: 2, Informative

    If you want a real Open Source emulator that's complete enough to run Windows XP (and do so with usable speed), QEmu is your only option. Currently it simulates a Pentium, Vesa fgramebuffer video card, and ISA NE2000 NIC.

    Check out this dude's blog for screenshots of QEmu running Win2K.

    Combine it with a copy of x86 glibc and a recent Crossover, you can use it to run Office XP for Windows on Linux on a Mac. Scary :^).

    Go get it from the Qemu Site.

  18. Recommendations by Anonymous Coward · · Score: 0
    I doubt that one person can do this.

    Having said that, I would do it in this order (some of this I've already done myself):

    1. Write a small OS with 32-bit protected mode code running at both kernel privilege (ring 0) and user privilege (ring 3)
    2. Extend your small OS so it can call 16-bit protected mode code such as the VBE 3.x video BIOS, PnP BIOS, or APM BIOS
    3. Turn your OS into a DOS extender by adding support for virtual- 8086 (V86) mode. With this code, it should be possible to call 16-bit real-mode interrupts from 32-bit protected mode code: INT 10h (video), INT 16h (keyboard), and/or INT 13h (disk I/O)
    4. Make an OS that supports multiple DOS sessions. A major part of this is virtualizing the hardware: 8237 DMA chip, 8253 timer chip, 8259 interrupt chips, 8042 keyboard controller chip, etc.
      The result should be like a primitive version of Windows 3.0, or maybe like Desqview, or maybe like this OS
    5. Add support for 16- and 32-bit DPMI programs to your multi-DOS OS. At this point, you should be able to run real-mode and DPMI programs in your virtual environment (like DOSBOX).
    6. Virtualizing 32-bit ring 0 code: x86 wasn't designed for this. You can run ring 0 code at ring 3 but you must trap all instructions that would give away the deception. This includes the unprivileged load/store segment register instructions! As hard as steps 1-5 were, this is obviously the hardest part. Maybe demand-load all pages of the virtualized program and, when the page is loaded, do a partial dis-assembly to locate unprivileged instructions that need to be virtualized. You might need something like the functionality of the IDA disassembler, which can generate a complete call tree of the program from the entry point.

    So, Josh. Do you still feel like doing this?

  19. Thanks for all the replies! by josh42 · · Score: 1
    Thanks for everyone's insightful, interesting, and on-topic posts. To address some questions that were posed...

    "Why are you doing this?" Because I want to learn more about how the x86 works, and I want to provide something that doesn't yet exist. "The Pentium is not virtualizable. You won't learn about virtualization in general, you'll learn about the Pentium." Great! :-) "Don't make a new project. Work on XYZ." I don't know anything about the internals of other projects, and I want to do something unique.

    To clarify: I'm wondering about virtualization because I'm working on a self-hosted VM monitor. (Think "ESX Server Lite".) It will be Open Source, of course, once I have anything remotely usable. (So far all I've done is set up page tables, IDT, IRQs, and use the timer interrupt to cycle the character in the upper-left corner.) Tentative name: XenOS. (Don't hold your breath...)

    (Are there any well-known patents on dynamic code recompilation in software? Just curious...)

    P.S. I'm using a virtualization thingy (VMware Workstation) to develop a virtualization thingy. Ironic.