Xen Hacker Interviewed

Drawoc Suomynona writes "The Xen virtual monitor is a new generation virtualization software that enable running multiple OSes at the same time with unprecedented level of performances. Manuel Bouyer was recently interviewed about his work porting Xen to the NetBSD operating system. The interview touches on why some consider Xen to be so good, how hard it is to integrate such a software package into an OS, and more."

Comments

Re:In other news...

[Anonymous Coward]

>Linux is STILL for fags.

That's right, real men use BSD.

Re:OS X ?

[simcop2387]

no, Xen requires changes be made to the kernel of the guest operating system, this can't be done with Windows, it could possibly be done with OSX though

Re:Can someone explain

[eggoeater]

Your answer and more:

http://en.wikipedia.org/wiki/Xen_(virtual_machine_ monitor)

Re:Can someone explain

[HitScan]

As I understand, vmware does do some limited emulation, at least VGA and Network cards. Xen instead traps all attempts to access the real devices in the machine and schedules them so that each operating system still thinks that they have full access to all of the real equipment. This requires some special kernel hooks, and that's why things like Windows and OS X aren't fully supported.

Also, I've seen this story in at least 3 places and I don't think it's right to say anyone ported Xen to NetBSD, NetBSD was updated (It's not exactly a "port") to take advantage of Xen features. It's possible that patches were sent to the Xen team to make things work more smoothly, but it's hardly porting.

Xen vs. jails

[nacturation]

Anyone know Xen compares to jails in, say, FreeBSD? I've managed to setup a jail before where you do a "make world DESTDIR=/jaildir" and then do a jail on that directory, which gives someone the appearance of their own entire operating system. Is Xen similar to this, but allowing for many different operating systems rather than just another instance of the same one?

Also, glad to see the BSD section is at least still around. I can't seem to get it to show up on the Sections list, regardless of how I set it up.

Re:Xen vs. jails

[LLuthor]

BSD jails share the kernel, Xen systems have their own kernels (patched versions of the actual Linux/BSD/etc kernels) and communicate with the host kernel through the vizor. Xen can run multiple different kernels and also support all the features such as advanced routing and access to host hardware.

Jails are lighter/faster but less secure (a kernel exploit in a jail will root the whole system).

Re:Xen vs. jails

[WindBourne]

The other major issue is that a jail is basically a chroot with some needed access to certain files. It will be one version of BSD / physical machine. In fact, jails are probably closer to UML on linux, then to Xen.

In contrast, Xen can run multiple instances of different OS. That is, on one physical box, you could run Suse, Mandrake, Redhat, NetBSD, and even Windows. So Xen is closer to vmware than to jails.

Xen on Windows

[Aladrin]

I see a ton of comments about not being able to use Xen to run Windows inside Linux, but no information the other way around...

Can Xen run Linux apps on my Windows installation? I am currently using Cygwin for that, and it's working okay, but some of my favorite apps are being run through SSH from my linux box to make all this happen.

I do too much in Windows to even dual-boot the system... I'd spend as much time booting as I would working/playing.

Re:Xen on Windows

[JoeGTN1]

CoLinux is a wonderful thing. http://www.colinux.org/

Re:Xen on Windows

[ocelotbob]

As modifications have to be made to the host OS to be a Xen host as well, I doubt windows will ever be a Xen host; MS is too headstrong with their own virtualization software. However, you can run coLinux, however, you will run into some performance problems.

Re:Xen on Windows

[hawicz]

You're not understanding how Xen works. Xen doesn't let you run one OS inside another, they run side by side, almost as if they were two processes.

Running OSes inside of Xen and running MS Windows inside of Linux are two completely different things. If you can run MS Windows inside of Linux, whether or not that Linux kernel is running inside of Xen probably won't matter, since for that to work at all you probably have to trap any protected instructions and emulate them. Whether the emulation is implemented using actual ring 0 instructions or Xen hypervisor calls should be irrelevant.

However, you _can_ run Windows inside Xen, and people have done so. It's difficult to do because you need to manage to get a Windows source license and build your own copy with the necessary modifications, but not impossible.

Re:Xen on Windows

[TheRaven64]

You don't run Xen on Linux, exactly. You run Xen-and-a-modified-version-of-your-favourite-kerne l on your hardware, and then run other (modified) kernels on that. Running in domain 0 - the domain with access to the real hardware - actually requires more modification to the kernel, not less, so there is very little chance of getting Windows to run Xen.

Re:Xen on Windows

[SillyNickName4me]

You could say that XEN itself is the 'host OS' abeit a very tiny one.

Virtual machines on XEN are called 'domains', and besides using virtual devices, they can also provide them.

Normally, domain 0 is responsible for providing almost all virtual devices (networking, disks etc). This may give the illusion that what runs in domain 0 is the host OS, but it is not, it is just another 'virtual machine', and while it is normal for domain 0 to do this, any domain running the proper kernel can provide devices to XEN.

Re:Xen on Windows

[ozmanjusri]

MS is too headstrong with their own virtualization software.

Microsoft are planning to release their own Hypervisor next year, and you're right, their version will be built into the Windows Server product.
http://news.com.com/Microsoft+hypervisor+plan+take s+shape/2100-1016_3-5735876.html

Re:Xen on Windows

[tinkertim]

You are correct and more so than you'd think. Xen provides true isolation of its dom-u's (user VM's). The xen hypervisor is most likely some of the most efficient code ever released to the open source community.

Xen layering and management allows you to do tons of stuff, I'm already doing SSI clusters on single machines. Xen + Win2k3 has been accomplished. This with CVIP / HA-LVS all running on one nic. Slice a high end p4 into a 6 + 1 (x 128) MB cluster of isolated servers. Its truly HA in a box, and very very simple.

The reason they call it a hypervisor is just that, its a step above a supervisory process. On VT enabled platforms (The new P4's / AMD's) you really start to see what xen can do without the bottlenecks of processor architecture.

Personally I think the ease of clustering is more important (and useful to the internet at large) than the ability to play with Windows stabalized under Linux. (I love saying that knowing its actually happened hehehehhe).

I can also say NetBSD does *very* well under xen.

Here's a really cool example config of how xen could slice up a high end dual xeon.

Assume .. Dual xeon .. 3.2 .. 4GB Registered 2x 250 GB SATA disks (one of my labs)

2 nics at 1000 MBPS, Connected to a gig-e switch. 100 MBPS x2 uplinks from 13 blended carriers. Basically, the average server you lease at any datacenter. Remember, you don't ever get to physically touch them. Xen is easy enough to install without needing local access.

You setup 2 smaller (maybe 256 MB each) netbsd firewalls , do some traffic shaping if you want. From there, you toss it over to an OpenSSI / Debian cluster running on the same machine.

Here's the really cool part. The bsd machines can talk to dom-0 and tell it when its time to drop nodes or add nodes, or make nodes bigger.

Need more servers? Simple . Xen them and load the ssi node image via pxe / etherboot.

Its very very easy then to setup the bridging needed to get a working cvip configuration and start weighting ports. So now you have 2 failover netbsd front end routers , failover LAMP and failover nics. Stick those SATA's in RAID1 and your only single point of faliure becomes your power supply or something going horribly wrong on domain 0. At the price it costs for those servers, you can afford 2 and pay under 500 bucks for the whole shebang if you lease them. Buying outright and co-locating is the best way. Or if your one of the fortunates with fiber coming into your own building ...

Now toss xen3 in there and you have yourself a win2k3 setup hosting your certificate authoirty, snaps, etc. bring it all into AD if you want. Its a networking "magic bag".

I'm just scratching the surface. These Guys Have a really, really useful wiki, as well as some "unofficial" Debian install packages. Your average Linux geek could get it going quickly.

Keep your eyes on Xen. Its going to do good things for everyone - and its going to push commercial equals to .. well .. be more equal. Right now (afaic) Xen tips the scales in its direction.

Windows is just one of the marvels folks. Look at the big picture. Some of us have been screaming Xen for a while now .. so its sort of a triumph to see it finally getting a larger following :) Virtuozzo just cant *touch* it.

Off the soapbox. Hope someone found this useful. It took an awful long time to type. Course would help if I wasn't eating messy food ..

Re:Xen on Windows

[TallMatthew]

The reason they call it a hypervisor is just that, its a step above a supervisory process. On VT enabled platforms (The new P4's / AMD's) you really start to see what xen can do without the bottlenecks of processor architecture.

I would think your bottlenecks would be constant context switches on your proc, cache misses on your virtual memory and seeks within your RAID (at the "hypervisor" level). No matter how good your top-level kernel delegates, it's still a level of indirection before control is passed to the kernels within your virtual machines (who obviously do their own prioritization). If you're running a cluster within a box, this constant delegating and shuffling of data from disparate elements to disparate kernels with disparate priorities would require constant and massive data shifts within the host machine and bring said box to a crawl, wouldn't it?

Re:Xen on Windows

[tinkertim]

You've basically just described what could happen to any processor (no matter how conjoined or how many) that could not return at or greater than the rate it was being fed, which is the point of using Xen in the first place :) I refer to it as 'meltdown' however.

You also have to keep in mind that most clusters are application specific.

With Xen and SSI you have two things that both do a very good job of :

1 - Replace "dumb" round robin load balanced racks (it makes a very good load balancer)

2 - Isolating applications (nevermind the os we're talking about a single image)

I'm not going to go into number one because its obvious (or is to who I'm replying to).

Lets look closer at #2. I'd like to (for demonstration) use as an example the vast number of people using an open source application on their website powered by Apache, PHP and MySQL.

I'd also like to call attention to the fact that commonly those aren't the only 3 malloc()'ing hogs running on any given single server. In fact you'll find most public services running in one place. This means a mailer (exim for this example), Spam Assassin, Clam AV (if they care about their mailbox), MySQL, SMTP bandwidth logging generally using MySQL, SSH, most likely POP and IMAP. Eh, almost forgot DNS but bind is pretty small. Now they're all figting for cache, while trivial system processes live happily in dentry and watch the public ones choke to death. Xen helps you stop this.

Imagine 300 http sessions (lets say some chat program mandated session keep alives), now someone rolls in with a brute force spam attack. There goes exim, spam assassin and clam AV.

Pretty soon things just stop forking and said server needs its diaper changed. So what you described is also what most people have existing.

Now take a look at any 20 places selling co-located servers , or leasing them. You've got about $200 - $300 a month you can spend. Your site was a hobby and now its a kick in your wallet. You'll find a nice Dual Xeon 3.2 (even a 2.8 would work) and you can get a few nics and 4 GB registered RAM.

You can, then with Xen and OpenSSI solve your problem, isolate your services, make some of them highly available and you (can) do it on a single platform and increase its capacity drastically. We have a few things at play :

1 - Xen's routing is very , very fast. That coupled with a sensible CVIP configuration can and will direct traffic as well as most medium line load balancers. I'm not talking about your $50k models that let you shape and direct down to the most miniscule trait of the session.. and I'm not talking about a cheapo. I'm also not calling out anything by brandname.. but I think you can relate for purposes of banter.

2 - You can't (and should not) run one of these from one physical ethernet device. While you don't need to give each node a seperate (real) nic, you really should split things up. By doing so you're freeing up kernel resources to do other things (like direct traffic avoiding I/O bottlenecks).

3 - You need to really play with your kernels. You really need to ensure you are taking advantage of your either (SATA) or preferably (SCSI) disks.

4 - You need to use sensible applications that interact nicely with your sql server, and (as you pointed out) have a very good understanding of Linux and its I/O. Be smart.. use flatfiles when you can (in other words plan your cluster).

5 - You can use xen in a more conventional setup too :) Just allocate 99% of any given server to your etherboot dom-u. iscsi / good gig-e nics and short copper runs to medium grade switches do this quite well.

So should Ebay fire one up today? No .. that would be as dumb as driving on your spare tire to look cool. Can your forum, e-commerce shop or php based game stay up and running? Sure :)

But what I just typed is several options available to site owners who 6 months ago only had much more expensive options.

Migration

[Stoned4Life]

"Xen also offers virtual machine migration, where you freeze a Xen guest, move it to another machine and resume it there ... This also means that a similar environment for the guest has to exist on the remote system."

And if a similar environment is not available when it is moved, what happens to the state of the user? Would the hardware in use when the state is saved have to be exactly alike on the target machine?

Also, is the information retained on the backup until the full migration is completed and then deleted, or is deleting the backup during the migration optional, leaving a "frozen" and "restorable" state on the server? Is that a security risk if the workstation is compromised?

Re:Migration

[dmp123]

Xen provides a hardware virtualisation layer, so the images don't talk to the hardware directly, but talk through the Xen hypervisor. Therefore, the Xen hosting on the remote system will provide exactly the same 'hardware' as far as the image is concerned.

David

Re:Migration

[kscguru]

Xen doesn't do graphics cards (at this point, everyone - even VMware - virtualizes graphics cards). Network cards, you only get via a virtualized Xen network adapter with Xen drivers, so those will remain the same. Nobody does passthrough hardware - nobody.

You will find processor differences though. Move from AMD to Intel, or drop SSE extensions or some such, and things will break.

Now THIS is cool

[ScrewMaster]

Xen virtual machines can be "live migrated" between physical hosts without stopping them. During this procedure, the memory of the virtual machine is iteratively copied to the destination without stopping its execution. A very brief stoppage of around 60-300 ms is required to perform final synchronisation before the virtual machine begins executing at its final destination, providing an illusion of seamless migration. Similar technology is used to suspend running virtual machines to disk and switch to another virtual machine, and resume the first virtual machine at a later date.

(Quote from Wikipedia)

Reminds of when I was watching the old Max Headroom show, and Max would shuffle himself off of one monitor onto a display on a portable "processing unit" and somebody would pick him up and carry him away.

Re:OS X ?

[MBCook]

My understanding is that Xen does not require the guest OS to be changed if the hardware supports virtualization (Vanderpool or Pacifica, depending on your chip maker). That means that with the right chip (I'm not sure if Core Duo has it or not) you could run OS X as a guest OS (I assume the host OS still needs to have support, which may be done with a simple application running as root).

VMWare emulates standard hardware

[Anonymous Coward]

so guest OSes that support that hardware can run with standard drivers. Xen creates a synthetic virtual device that's easier and more efficient to emulate than standard hardware. The downside is that there usually aren't any drivers for these virtual devices on most OSes since no one's written them yet. So if you go on Apple's web site and look for supported video hardware, you probably won't see Xen virtual video device. The other way of looking at it is Xen is the device maker and is supplying the driver directly rather than through Apple. And device makers don't always supply drivers for all OSes, at leat right away.

Re:In other news...

[kadathseeker]

What? You like BDSM? Thanks for sharing, but... uhh... no thanks. Well, it probably is VERY manly.

Re:What I want from virtualization

[Anthony Liguori]

What I want from OS virtualization is to be able to run one guest OS on multiple hosts for redundency.

Then what you want is Marathon style lock-style execution. It's a terribly hard problem because you have to make all software run deterministically (timers and IO events on both machines have to occur in the *exact* same moments in execution).

Regards,

Anthony Liguori

What about Virtualization?

[horacerumpole]

I think you should update your information about this.

Xen 3.0 on the newer Intel/AMD chips should be able to run Windows (or any other OS) without modification to the hosted OS.

Re:What about Virtualization?

[ocelotbob]

True, but he was asking about using windows as the host, which will not be possible unless you shell out the money to purchase Longhorn Server. It's doubtful that MS will provide the hosting hooks into the desktop versions of Longhorn

Try VMWare ESX Server 3

[supachupa]

VMWare ESX Server 3 (currently in beta) will probably get as close as you can to having live fault tolarant servers. By sharing a storage such as SAN or iSCSI, it is able to instantly take over when it detects the failure of another physical box and should not lose any sessions or data. The other cool thing is that it can dynamically move a running virtual machine to a less loaded physical server. Still, it is not running two virtual machines simultaneously like you've described.

Re:VMWare Player...

[thehunger]

You're right. Any you'll be glad to know that VMware has decided to provide a updated and FREE version of its GSX Server. This will allow you to CREATE virtual machines as well as run several VMs timultaneously. Also it can do SMP and 64-bit emulation, so you'll be able to install a 64-bit multi-processor version of an OS on your current 32-bit single processor computer...

  It's called VMware Server and is in beta now.

However, it's not -entirely- free: VMware will charge for support, and there's no guarantee that updates and patches will be available for non-paying customers.

Re:Can someone explain

[TheUser0x58]

the main reason why Windows and Mac OS X are not supported in older Xen systems is because Xen allows guest OSes to run without trying to intercept privileged instructions. Thus, operating systems which haven't been "ported" to Xen wont run because they expect these instructions to work (but they dont). VMWare "ports" guest OSes at runtime, actively changing these privileged instructions in the guest OS, so that it doesnt need to be modified beforehand by the user.

And also, the Cambridge guys did come up with a Windows XP port for their own academic purposes, but they obviously cant release it for licensing reasons, etc.

Re:Can someone explain

[stedo]

I don't think it's right to say anyone ported Xen to NetBSD

You're right, nobody ported Xen to NetBSD. That's not how Xen works. What happened was someone ported NetBSD to Xen. Instead of this new version of NetBSD accessing hardware, it asks Xen to do it. This required no modification of Xen, just modification of the NetBSD kernel to avoid accessing hardware directly. It's comparable with porting NetBSD to a new chip architecture.