Desperately Seeking Xen

AlexGr sends us to an excellent article on the state of Xen by Jeff Gould (Peerstone Research). He concludes that the virtualization technology has some maturing to do and will face increasing competition for the privilege of taking on VMWare. Quoting: "What's going on with Xen, the open source hypervisor that was supposed to give VMware a run for its money? I can't remember how many IT trade press articles, blog posts and vendor white papers I've read about Xen in the last few years... The vast majority of those articles — including a few I've written myself — take it as an article of faith that Xen's paravirtualizing technical approach and open source business model are inherently superior to the closed source alternatives from VMware or Microsoft."

Comments

Need a special processor

[flyingfsck]

Xen only works with specific hardware which I don't have. Sooooo, back to VMware.

Re:Need a special processor

[Mostly a lurker]

It is true that Xen requires special hardware to legally run MS Windows. It is also better for performance, generally, to have such hardware. However, there is nothing stopping you from running Xen on pretty much any computer you are likely to own as long as the VMs are Linux based.

Re:Need a special processor

[mwilliamson]

VirtualBox doesn't require any special hardware.

Re:Need a special processor

[friedman101]

VirtualBox (http://www.virtualbox.com) has an open source edition of their virtualization software which is easy to use, cross platform, and works well on my machine. It can also read vmware disk images which should make transitioning pretty painless. The closed source (but still free as in beer) version also supports USB and simple file sharing between guest and host. I'm currently using it to run Windows XP inside my Vista machine to take care of compatibility issues.

Re:Need a special processor

[flyingfsck]

Yup, I need the ability to run Win 2003 Server on a VM and the only way I could make that work was with VMware. Win 2003 crashes Qemu and Virtual Box and needs a special processor for Xen.

One funny I found with VMware Server is that it cannot run RedHat RHEL version 5. The install proceeds partly then screws up.

Each and every VM system has a bunch of quirks, so one has to try the lot to find a good one for your purpose.

Re:Need a special processor

[TheRaven64]

However, there is nothing stopping you from running Xen on pretty much any computer you are likely to own as long as the VMs are Linux based You mean 'Free Software,' rather than 'Linux based.' You can run NetBSD, FreeBSD, Linux, OpenSolaris, Minix or Plan 9 (maybe some others I've forgotten) as paravirtualised domU guests with no special hardware. You can also run OpenSolaris or NetBSD instead of Linux in domain 0, if you don't feel like running Linux. Which makes me wonder slightly what this is doing in the 'Linux' category...

Re:Need a special processor

[tropicdog]

I think you mean VirtualBox.ORG, not com.

Re:Need a special processor

[rbanffy]

If the VMs are all Linux-based, a single-kernel approach like Virtuozzo/OpenVZ makes more sense than Xen.

Xen is nice if you need to run many different kernels or different OSs tailored for the Xen way and sure keep things more isolated in case of a security breach, but the added walls are somewhat inconvenient for managing the virtual servers. I encountered many inconveniences in managing an 8 VM setup under Xen, but managing a similar setup under OpenVZ is a breeze.

vm ware

[Anonymous Coward]

vm ware is feature king at the moment - but competition is a good thing.

If you want to run multiple linux instances on the cheap then xen is the way to go at the moment.

Re:vm ware

[Anarke_Incarnate]

Virtual Iron has some features,that are important that VMware does not :)

Re:vm ware

[Em Ellel]

vm ware is feature king at the moment - but competition is a good thing.

If you want to run multiple linux instances on the cheap then xen is the way to go at the moment. Not that I disagree that OSS and competition is a good thing but what am I missing here? Last I checked VMWare Server product was free - so how much cheaper than free can you get?

-Em

Re:vm ware

[stoolpigeon]

server (free) and esx (expensive (to me anyway)) are very different products, and many of the really attractive features exist only in esx.

Re:vm ware

[Em Ellel]

Virtual Iron has some features,that are important that VMware does not :) Can you expand on this? I never even heard of Virtual Iron before just now, what useful features does it have over XEN/VMWare?

Thanks

-Em

Re:vm ware

[Per Wigren]

If you want to run multiple linux instances on the cheap then xen is the way to go at the moment. Except that OpenVZ is a better way to go in that case. If you are only going to run multiple instances of Linux, with OpenVZ you don't need to preallocate a fixed amount of memory for each VM, the root filesystem can be a subdirectory of the root OS instance's filesystem, among many other things. It can do just about everything that XEN can do, including live migration to other physical nodes.

Re:vm ware

[CastrTroy]

Could you outline some of the features on ESX that you find really attractive? As a home user, I find that VMWare server fits just about all my needs.

Re:vm ware

[gtrubetskoy]

Except that OpenVZ is a better way to go in that case. For fairness it should be mentioned that aside from OpenVZ there is also Linux VServer which does a few things better than OpenVZ (though OpenVZ does some things it does not). Our preference has always been VServer, it's a well-run project with emphasis on quality and well thought through design rather than quantity.

Re:vm ware

[Em Ellel]

The major things that makes ESX attractive as far as I know all have to do with Enterprise usage - i.e. bare metal hypervisor (self contained "Host" OS) and ability to live transfer a VM from one server to another via shared storage without shutting down.

Can't see this impacting a home user, unless I am missing something.

-Em

Re:vm ware

[Anarke_Incarnate]

Virtual Iron uses Xen but does not do para virtualization. It supports 8 CPUs per guest and is a "bare metal" VM hypervisor (does not require a host OS).

It also comes with the ability to move virtual machines from host to host (based on pools of resources) either by some threshold being met (say, 75% CPU utilization on a Dell PowerEdge server moves it to an HP Proliant which is only 45% utilized ) without the virtual guest OS ever skipping a beat. Vmware does this with VMotion, but that is an add on package. Virtual Iron also supports raw devices presented from a SAN and can boot the "Hypervisor OS" from SAN as well. It supports Microsoft's disk format and there are free tools that can convert VMware disk images into that format so any Virtual Appliance built for VMware can be used in Virtual Iron (and no, I don't work for them).

However, because it uses Xen and does not do para virtualization, AMD-V or Intel VT must be available on the CPU being used. My plan was to use VI on Sunfire X4600 M2s with 16 cores available and the capacity to have 256GB RAM and 4 built in GigE (with slots enough for more GigE and Fibre Channel).

Check out their demo on their website

Re:vm ware

[b1ufox]

Ok, Virtual Iron is exclusively for Vt-x and SVM enabled processors.

This means it makes use of the hardware assisted virtualization provided by these two techniques by Intel and AMD respectively.

Xen on the otherhand takes a paraivirtualization approach. It includes a hypervisor which shields the hardware access from guest domains(unless you are running a driver domain).

So Virtual Iron can run unmodified guests from its first release, as opposite to Xen which started in era when there were no SVM and Vtx capable processors

So, you cannont strictly say adavntage over Xen.

OTOH Xen though predominantly a PV hypervisor can still make use of VTx and SVM processors, to run unmodified Guest Operating Systems but performance hit is there.

So, Xen is a great product which needs a lot of grooming before it can be considered a dependable virtualisation solution.

It is better than VmWare in the sense of end usability, and that means throughput, latency and responsiveness.

Re:vm ware

[stoolpigeon]

Right - there is an intersection here of two completely different user groups - I belong to both myself. I use vmware server on my personal/work machine - so that I can have xp available for a couple windows only apps that I must use. In the datacenter we use a mix of aix virtualization on p5s and esx. virtualization in the datacenter is bringing our organization a number of benefits that server can't provide. My understanding is that one feature is moving virtual machines from one host to another on the fly. There are probably more - but I'm a dba so I'm at the edges of this stuff.

Re:vm ware

[caluml]

Commas are, like violence - you, can always use, more to solve a problem.

Re:vm ware

[neonux]

Linux VServer is also quite easier to configure/maintain compared to OpenVZ imho :)

Re:vm ware

[Anarke_Incarnate]

The benefit are enterprise level features that exist right now that Xen does not have in the F/OSS version. The "Commercial" version will have those features very soon, if not already. Many large companies WILL NOT allow modified kernels or modified guests in general to be released into production. The lack of support for older CPUs that do not have built-in virtualization is the reason I cannot use Virtual Iron just yet for my company, but I am pushing them towards different machines for the future. I see that as an uphill battle because they are on a blade vendor that does not support those chips right now. I wish we could just dump that in general and just go to Sunfire X4600s.

Re:vm ware

[a1mint]

I use vserver all the time, I like it.

I wish the /etc/vserver directory and the excessive symbolic links weren't all over the place though. The configuring I find a little convoluted.

Re:vm ware

[Tony Hoyle]

ESX is a bitch to get working. We got a demo, set up a machine with hardware entirely on VMWare's 'approved' list - and it wouldn't install because it was missing drivers. VMWare couldn't help because we hadn't parted with any money yet... we went with RHEL w/ the free VMWare server which 'just works' (xen doesn't have the I/O performance to be deployed seriously yet).

Re:vm ware

[BagOBones]

Who is your hardware vendor? Out of the box ESX ran on both the server and the SAN we where using with NO extra drivers.

Re:vm ware

[CastrTroy]

Transfering a live server could be useful at time I guess, although like you said, probably not necessary for the home user. Also, I was aware of the "bare metal hypervisor", which I guess probably has quite a few advantages, since just the host OS can eat up a lot of the machine resources. How bare of a Linux machine can you run VMWare server anyway? You can run Linux itself with very little resources, It should be possible to create a distro that's cut down to just the stuff necessary to run VMWare.

Re:vm ware

[Quikah]

No SMP on windows guests, super flaky networking, fragile configuration DB, no real CLI management tools. I am not sold on these features myself...

I fully admit, I am sure at least 50% of my problems is due to my unfamiliarity with the product, but I have been able to setup Xen and ESX with the same lack familiarity with little trouble, I have never used a buggier or less user-friendly piece of software than VI, good lord.

Timely for me!

[grub]

Just yesterday I wrote in my journal asking for VM advice. Sadly it seems none support 3D cards very well.

Re:Timely for me!

[Dutchie]

True. This is a major problem IMO. Once upon a time I had hoped that I could run VM's *and* have full 3D support, both in a Windows VM and a Linux VM.

Turns out that 3D accelerated is not an option right now, but Xen was at that time working on something that could (given the right hardware, which at the time was only high end IBM mobo's) isolate PCI cards completely.

That way, you could have two graphics boards in your system, and when Xen starts up it could assign one graphics board to, for example, a Windows VM which could then run 3D accelerated drivers with it's native drivers.

I wouldn't quite give up on Xen yet :)

Re:Timely for me!

[stoolpigeon]

I just think at this stage in the game, as the article implies, the primary focus is the datacenter. And 3d just doesn't matter there in the vast majority of the cases. Certainly not enough to make 3d a top priority.

Re:Timely for me!

[mrchaotica]

IIRC, the latest (i.e., so-called "Direct3D 10 compatible") graphics cards have MMUs, which would (theoretically) allow multiple OSs to share the card in the same way that they can currently share the CPU.

Re:Timely for me!

[TheRaven64]

There was some interesting work presented at the XenSummit which should be making it into the main tree eventually on 3D support in Windows. The idea was that the memory layout was adjusted slightly so that Xen and dom0 lived nearer the top, and Windows in an HVM domain lived at the bottom. This allowed Windows to use existing 3D drivers, without having to do any address translation when performing DMAs (not an issue if you have an IOMMU, but hardly anyone does yet).

The problem with giving access to hardware to guests at the moment is that without an IOMMU, any DMA request the driver issues will read or write memory from a physical address indicated by the driver. In a virtual machine, what the driver thinks is a physical address is actually a virtual address. This means a DMA request will read from or write to an arbitrary memory location. By putting the HVM guest at the start of memory, this translation is the identity function, so the driver will work. The only downside is that you lose protection from other domains; a malicious driver can still damage your other VMs or even the hypervisor.

Re:Timely for me!

[MajinBlayze]

The problem with Hardware acceleration in VMS is fairly straightforward. The driver sends information such as 'use bitmap located at pos x in memory' The way memory mapping works, the VM might be given a chunk of memory (i.e. positions 100 to 200) and sees this as 000 to 100. for the VM, x = 010. When the card tries to access that memory, it's memory that might be assigned to a different VM, and thus garbage. Unfortunately, this generally requires the cooperation of the drivers.

fortunately, VirtualGL is working on a generic fix, although it's still immature.

Re:Timely for me!

[MajinBlayze]

That's not true; Look at parallels for the Mac. This could be an important part of offering windows apps for the Mac. Additionally, I'm currently in the process of setting up Xen at home. I'm buying a new computer in the not-so-distant future that will replace both my Gentoo box and my Wife's Windows computer. The theory is that I can virtualize a few different environments so that my messing around doesn't interfere with her Windows environment. Plus I can get a stable desktop environment, a stable server environment, and test environments for both. Virtualization has a lot of potential in many areas outside the server farm. Ideally, I'd like to be able to run Vista/Aero on the VM, but that's still a ways off.

Re:Timely for me!

[MajinBlayze]

Any idea what this is called? how I can find it, or if there are unofficial patches yet?

Re:Timely for me!

[Anonymous Coward]

Another great reference that I ran across after posting here. This is really what I was looking for. Posting anon to not be karma whore.

VMGL [PDF warning]

Re:Timely for me!

[rbanffy]

I am a karma whore, you insensitive clod.

Re:Timely for me!

[Anonymous Coward]

you don't have to replace your gentoo box! you just emerge Xen!! (afaik :p)

I'm still thinking wether I wanna do that on my server though ...

Re:Timely for me!

[Anonymous Coward]

VMWare Server does not have an accelerated X server, true, but you can work around that (I have done this and It Works For Me):

Install <your linux distro> on host machine
Install vmware server on host
Install <your linux distro> on guest VM
Install OpenGL apps of your choice on guest VM
Start up X on your host machine
user@host:~$ ssh -X <guestVM> <OpenGLapp>

Advanced lesson: configure [KDM|XDM|GDM] on your host and guest to allow your host to log into your guest, thereby using all your guest's apps on your host's accelerated display.  This was *very slow* for me, but that was probably due to the fact that I was mounting my host's /home as /home on my guest, too.

Obviously this doesn't help you if your guest is Windows.  Just plunk down whatever the going rate is for VMWare Workstation ($299? I have no idea) and be done with it.

Re:Timely for me!

[TheRaven64]

You can read the slides and if try contacting the author if you're interested.

VirtualBox performance

[mwilliamson]

It seems that VirtualBox.org's product, fully virtualizing a copy of XP on my non-VT machine under a linux host OS, totally runs circles around Xen even on VT hardware as far as performance is concerned. Integration into the host enviroment is also quite beautiful. Why is there seldom a mention of VirtualBox in this arena?

Re:VirtualBox performance

[jsailor]

I can't comment on why VirtualBox doesn't get more press, but I can confirm that I've had very good results using VirtualBox 1.3.x on fairly low powered machines. My guess is that it gets lumped under QEMU when comparisons are made.

Re:VirtualBox performance

[drinkypoo]

Why is there seldom a mention of VirtualBox in this arena?

I just installed virtualbox on Ubuntu Feisty to see what your fuss was about. I tried to install Windows 2000 in a VM and VirtualBox wouldn't let me type F8 to accept the license. No idea whose fault that is, but speaking for myself I can say only that vmware server 'just works' and thus I have no reason to use virtualbox, which does the same things but not as well.

Re:VirtualBox performance

[timeOday]

IMHO VMWare took a huge chunk out of the open source alternatives when they released the Player and Server for free (as in beer).

Re:VirtualBox performance

[Anthony Liguori]

It seems that VirtualBox.org's product, fully virtualizing a copy of XP on my non-VT machine under a linux host OS, totally runs circles around Xen even on VT hardware as far as performance is concerned. Integration into the host enviroment is also quite beautiful. Why is there seldom a mention of VirtualBox in this arena?

Because the mechanism they use to virtualize on non-VT/SVM systems is incorrect. Since a guest will likely eventually crash, it's not interesting for anything but hobbyists. They technique they used is fundamentally broken and will never be fixable so it's just not on people's radar. The same applies to kqemu.

As I understand it, when VT/SVM is present, VirtualBox does behave correctly. The reason it isn't interesting there is that the code is horrific and will never make it into the upstream kernel.

Re:VirtualBox performance

[mwilliamson]

Does that same argument apply to VMWare too then? (for their product that allows you to fully virtualize non-modified operating systems on non VT/SVM hardware)

Re:VirtualBox performance

[Anonymous Coward]

Have you tried clicking in the VM window to capture keyboard input? My experience has been that this doesn't always happen automatically when starting the VM.

Re:VirtualBox performance

[nailBnny]

I had a similar problem. It was caused by a Beryl shortcut for their "expose" feature being linked to the F8 key. I just turn off Beryl during that part of a windows installation. You can change the shortcut in Beryl Settings Manager > Shortcuts > Keyboard and Mouse > Scale > Bindings > Initiate Window Picker for All Workspaces.

Re:VirtualBox performance

[Anonymous Coward]

You have to click on the virtual desktop in virtualbox to switch the focus to the vm, then f8 will work fine. Installed Windows Vista Ultimate on VirtualBox no problem.

Re:VirtualBox performance

[drinkypoo]

I clicked in a lot of places but none of them made f8 work fine. And f8 works in vmware when I use beryl so if that was the problem then it's still not my fault :P If it's a pain in the ass then it's more trouble with vmware and I'll wait for some revision before I screw with it again.

Re:VirtualBox performance

[Anthony Liguori]

Does that same argument apply to VMWare too then? (for their product that allows you to fully virtualize non-modified operating systems on non VT/SVM hardware)

No. VMware uses dynamic translation which is a complete mechanism. VirtualBox uses binary patching which severely violates correctness.

Nice Article

[stoolpigeon]

Not just for xen - but if you are interested in virtualization in general. Lots of links to many other products - open and closed. So if you aren't into xen, but still want to know about what is going on in this space (to some extent - they don't even touch the stuff IBM is doing really) then it's worth the time.

KVM vs. Xen

[aztechClanIII]

I am also a VMWare fan and was very happy when they released VMWare server for free a while back. I use VMware Server every day to run a Windows XP machine and Ubuntu Server in a virtual networked environment which I develop client/server apps on. It works awesome.. although I long for faster hardware emulation.

I'm rooting for KVM to take over, however it probally needs some time to settle down. Networking is harder to setup, and I've noticed graphical glitches in winXP.

Because....

[Ayanami Rei]

virtual box is basically QEMU with a much better KQEMU component that they developed on their own. This isn't very interesting because this is the same thing as VMWare or any other closed source Ring0-in-Ring1 emulation using polymorphic code.

what went wrong is

[advocate_one]

you can't run Windows using Xen as Microsoft won't let you recompile the windows kernel... and this ability of being able to run windows on Linux is one of the things Microsoft allows to be done with the blessed versions of Linux for corporate customers. Normal mortals can't do it and will never be able to do it.

Personally, I don't give a flying fig about being able to run Windows or windows programs on Linux... there isn't anything I want to do on windows that I can't do on Linux... (note the emphasis, I find everything I need in Linux...)

Re:what went wrong is

[stevey]

Not true.

If you have VT-capable hardware then you can run Windows under Xen. You do need the hardware to support it though, and that is a problem for some home users. Recent AMD and Intel chips have slightly differing VT support but both work.

I run Xen at home along with xen-tools (which I wrote) to easily create new Debian guests on demand. These are used for software testing, hacking, and general service isolation.

I think Xen is just now reaching "mainstream" in the sense that you don't have to be an early adoptor or major tinkerer to get it working. Now that distributions are including Xen kernels in their newer releases it really us available for all.

Re:what went wrong is

[pinkocommie]

I've been messing with XEN for a week or two, thought it would help me out with debugging and maintaining multiple systems without the need for them. Picked up an AMD SVM CPU and tried it, running into a wall. The XEN list serv wasn't much help nor were the log files. In comparison Virtual PC 07 / VMWare on Windows worked flawlessly on the first try... I'd still prefer using XEN and might take a stab at KVM but wasting a week just to get to the boot screen without success was a little painful. FYI I tried CentOS and WinXP

Re:what went wrong is

[Anonymous Coward]

Yes, but as he already told you, what you know is out of date. You can run an un-modified Windows on top of Xen provided your hardware supports the Intel or AMD VT extensions.

Re:what went wrong is

[postbigbang]

Wouldn't it be nice if you could get a Xen rootkit for Windows XP and/or Vista?

Perhaps it's a silly concept, but it could make work easier.

But then there's the graphics difficulties.... the need for a hyperthreading CPU....and there's no support of course, for a rootkit ;)>

Re:what went wrong is

[DataSpring]

In addition, you can achieve very good performance with Windows on Xen (utilizing VT/SVM processors) if you also install the Xen Paravirtualized drivers under Windows, once you have the base OS installed. You can get these drivers in the XenServer edition on the xensource.com site. (Actually, I think my coworker got the drivers running in the XenExpress edition.)

In any case, performance was significantly better with the PV drivers - in fact, when we first installed a copy of Windows under Xen, disk performance was so bad that a five-year old machine that is overloaded was serving up a website faster than the virtualized version of Windows. After we installed the PV drivers (and rebooted the Windows VM,) performance was near native (we have two of these servers, and were doing exhaustive tests on both at the same time using both benchmarking tools and real-life tests, one had Windows installed natively, one had Xen + Windows.)

After further testing, we were so impressed, we put it into production. On the Xen machine, there are two Windows guests and one Linux guest. Windows 1 hosts the Asp.NET website, Windows 2 hosts MS SQL Server 2005 Express, and the Linux VM hosts DNS services. There are real-world websites running on this platform and performance is great. Notice, I didn't say "adequate" or "good" - it really is great, and we plan to put four more servers into production with a similar type of configuration for various purposes.

Re:what went wrong is

[Tony Hoyle]

I hope you paid for that xenserver....

I've tried them and the performance still isn't great - xen seems to have bottlenecks on its network and disk I/O that are a result of using qemu to do it in software... the maximum net throughput even on PV is a fraction of a 100mb link let alone a gigabit one, and my old 486 firewall does faster disk access.

Re:what went wrong is

[Anonymous Coward]

On throughput tests of Win2k3 with the PV drivers, I've been able to max out a gigabit link. Compared to bare metal, there is a slight performance degradation.

I don't recall what the disk I/O performance was, but it was pretty good too.

Re:what went wrong is

[Dorkmunder]

I'm running Win2003 as a guest with the PV-enabled tools and getting way past the 100Mb speed for NIC I/O (I haven't pushed it to Gbit yet). I can't comment on the disk performance, though, as of yet as I haven't started running any disk-heavy apps on Xen yet. I would agree with the GP though in that I'm seeing excellent performance running three OS'es (two win2003 servers and a Debian Etch server) in Xen on a Dell Blade with the right processors, almost bare metal speed for what I'm doing. Again, I haven't been ready to throw a SQL or Emai type server at it yet but for the more processor\memory\IO heavy apps\servers I'm quite happy with the performance (and cost savings and energy savings, etc).

Re:what went wrong is

[rthille]

Any experience with Xen, NetBSD as Dom0 and Windows? I've got a NetBSD server that I'd like to be able to run Windows on, but VMWare on NetBSD seems out of date. I suppose I could use Linux for Dom0, run my server stuff on NetBSD as a DomU, but getting familiar with YAOS seems a pain...

Re:what went wrong is

[Anonymous Coward]

If you have VT-capable hardware
You mean like this?

Re:what went wrong is

[commonchaos]

Thanks for writing xen-tools! I'm very impressed by all the features it supports (LVM! Debootstrap!). The only downside to using xen-tools is that using VMWare becomes a very frustrating task.

Re:what went wrong is

[stevey]

:)

Re:what went wrong is

[Anonymous Coward]

What I really wish is that there were PV drivers for Windows running on Xen. There are for the paid versions, and the very limited Express version, but not the open source version. Since I have absolutely no interest in paying for support on something I can figure out myself, they're kind of shooting themselves in the foot here for full adoption.

Re:what went wrong is

[pintpusher]

second that! thanks for xen-tools. It makes my home server a breeze. I've got my local lan file-server/approx server as Dom0, firewall, apache server and mail server all running in DomU's with pciback hiding my public interface from everyone but my firewall. Its slick and easy. (idea totally stolen from shorewall, btw)

I know that running different uses in VM's is not real efficient, but my loads are low and I can tinker with parts of the system while my wife is still able to listen to her music. Never underestimate the trouble caused by "honey, I'm upgrading the mail server and I'll be taking the music/vid/photos offline for a while..." or the benefits of overhauling the firewall while still having access to the IMAPS/SMTP server available (even if nothing gets delivered for a while).

Host OS the one with better drivers

[interiot]

If I have to maintain two separate OS's, I'd rather have the outermost OS (host OS) be the one that has the best drivers, the most hardware support. Also, since very few virtualization solutions work with 3D gaming (and even the one that does, it still has large overheads I think), you want your host OS to be the one that has all the games. So, for my purposes anyway, I need Windows as the host OS, and Linux as the guest OS. Xen doesn't run under Windows, only Linux. So that leaves me with either commercial virtualization software, or a few open source projects that haven't matured yet (eg. coLinux).

(granted, having Windows on the outside makes your machine much less secure than the other way around, but personally, I'm more interested in having all my peripherals work the day they're released, and having all my games available)

Re:Host OS the one with better drivers

[jsailor]

My situation is different, but with the same requirement for a Windows host OS. VirtualBox fills this need quite nicely. The latest release enables the use of VMware disk images, but I haven't tried that. You may wish to try out VirtualBox.

Re:Host OS the one with better drivers

[interiot]

I'd prefer to be able to dual-boot directly into Windows or Linux (for when I want the fastest performance in Linux, and give it 100% of the RAM), and also be able to run that Linux installation inside of Windows. However, this requires the VM to support booting off a separate partition, and apparently VirtualBox doesn't support that. (yes, booting the same Linux setup under two very different sets of "hardware" has its challenges, but it is possible)

Fundamental performance issues

[Mostly a lurker]

Full virtualization, as used in Xen, VMware and VirtualBox, has performance issues that are not yet well understood, but thought to revolve around dramatically increased L2 cache misses. I am not aware that any changes are in the works to fully resolve this.

Operating system virtualization, as used for instance in OpenVZ has far better performance characteristics. This is the way to go at the moment for efficient and low cost data center support of Linux. The problem is that all virtual environments must be using the same kernel, which makes it less useful for software testing.

Re:Fundamental performance issues

[Courageous]

Full virtualization, as used in Xen, VMware and VirtualBox, has performance issues that are not yet well understood, but thought to revolve around dramatically increased L2 cache misses. I am not aware that any changes are in the works to fully resolve this.

Hmmmm. No, not really.

Performance problems with VMWare are almost universally associated with the added latency of the multiplexing/demultiplexing code that needs to be run to talk with shared I/O devices. This added latency in turn impacts bandwidth. "L2 cache misses" refers to CPU overhead, of which there is little to speak of in any of the above technologies (5% with VMWare, practically zero with Xen). CPU should be of no concern to you with any virtualization technology that I know of.

I've validated this time and time again with different benchmarks from SPEC (not just the CPU benches, but NFS and so forth, as well as OSDB, basic simultaneous FTP tests, and so on).

C//

Re:Fundamental performance issues

[Mostly a lurker]

I would be interested in knowing your assumptions, test streams and the hardware configuration. Several carefully executed benchmarks of real world scenarios that I have seen do not tally with what you are claiming. For instance, see Performance Evaluation of Virtualization Technologies for Server Consolidation and Diagnosing Performance Overheads in the Xen Virtual Machine Environment

Re:Fundamental performance issues

[Courageous]

20 Dell 1955 Blades; 16G ram; 70GB SAS 10K drive (one) on which ESX 3.0 is hosted (or a variety of Xen flavors); four gigabit ethernet controllers per blade; CISCO 4948 48 port switch, with 4 ethernets per blade bonded; CISCO 6504e core with Sup-32; Net App 3020 and 3050 for NFS and iSCSI; some EMC Clarion units, likewise.

For CPU we used SPEC CPU 2006 and score about 5-6ish % on VMWare as the same test done on those blades in hard metal. Xen is undiscernably different to the subjective eye than hard metal. I would have to break out large batch testing methodology and run the results through inferential statistics to conclude that there was a difference at all.

I/O is a different story.

The Xen performance claims and the VZ performance claims aren't really useful. They're theoretical. As in, "theoretically, we can stack 100 operating systems on this blade efficiently." Think about that. That's just plain nuts. I can't think of a real use case for that.

BTW, if you like OpenVZ, and have the right use case, the commercial Virtuozzo product ranks as the "best virtualization technology that no one has ever heard of" in my book. They really have their IT management story down pat.

C//

Using it...

[dmayle]

You can choose to believe the hype or not, as you wish, but I'm using Xen in my production environment, and it's simply fantastic. I've got friends with companies who are doing it as well, and it really changes how you think about administration.

Of course, there are some learning curves. For example, how you manage 3-7 servers is completely different from how you manage 20-30, even if they are all virtual. There's a lot more emphasis on system images, isolating functionality, reproducing configurations. On the other hand, dev environments are so much easier to build-up and tear down.

I just wish the OpenBSD port was in a usable state. The mercurial servers hosting it are often down, and even when they're up, I haven't been able to get a working kernel compiled from the sources (even after doing some of my own bugfixes). And last I saw on the Xen lists, Christoph Egger (the guy doing the OpenBSD port) submitted a security patch related to stack slamming, and the Xen guys were kind of like, "meh, security's not really a priority..."... Oh well, here's to keeping my fingers crossed

Re:Using it...

[cheshire_cqx]

I have a VPS ("slice") at Slichehost and it's the best thing since... well, you know.

Seriously, it may not be right for all applications, and things like Solaris' zones/containers are quite awesome (much more control over IO, fair share scheduling, etc.), I have one of those too at Joyent, but (like many things Linux) it seems to work, be fast, and get the job done at a great value.

I have seen people complain when they have an app that's IO bound and there's another slice with heavy IO needs--looks like IO contention isn't really managed by the hypervisor--but that's pretty rare.

Besides performance at a great price, you also get the benefits dmayle talks about: easy deployment (you get a choice of prebuilt distros to install), and backups/spanshots (both create and restore) are as easy as clicking a button in the control panel. You can also upsize or downsize your slice from the control panel, and be back online in minutes. Try that with a dedicated server!

desperado...

[fattmatt]

what is more desperate is the people that think these highly specialized OSS apps that are not competitive in terms of features/support are going gain any traction in the market place. ha.

The Problem With Xen

[Aeonite]

Is that Gordon Freeman ruined it.

Pointless?

[ichigo 2.0]

Then what's the point? What special applications does Linux have that require you to run it in a VM on Windows?

Usually people run Windows in a VM because they have some app that just doesn't exist on Linux, but that can hardly be the problem as most Linux apps are OSS and thus portable to Windows.

Re:Pointless?

[gbjbaanb]

I think you have that slightly wrong - when you say "Usually people run Windows in a VM", I think you meant to say "Usually people run Windows".

Some of them will take the plunge into the uncharted waters by running Linux in a VM so it won't trash their desktop settings, apps etc, ut make no illusion that the majority of people using computers are using Windows, not Linux.

Re:Pointless?

[ichigo 2.0]

Sorry to rain on your parade, but what I meant was that the reason people (who aren't on Windows) run Windows in a VM is because they need it for some app. Obviously people who already are on Windows have no need to run it in a VM.

Why not use a live-cd if one wants to test it without committing? The inherent slowness of VMs makes it difficult to ascertain how well Linux would run, and the lack of 3d-acceleration means that none of the shiny things that draw in new people will function.

Re:Pointless?

[CastrTroy]

I just can't wait till KDE4 is ready. Then I can have Amarok, Kopete, and most importantly Digikam running on windows. Hopefully it's stable, and runs well.

Re:Pointless?

[interiot]

that can hardly be the problem as most Linux apps are OSS and thus portable to Windows

True. However, if you work with a large number of open source apps, or even just a lot of Perl modules... Usually these were designed from the start to work under Linux. Yes, the more popular ones compile under both, but sometimes it's a pain, and the less popular ones simply won't compile without extra work.

Also, I just prefer Unix streams/forking/filesystem semantics over Windows. And sure, I use Cygwin/MinGW/etc. a lot, but I'd still take Debian/Ubuntu packages over them any day.

Re:Pointless?

[Wolfrider]

--Running a Linux Squid appliance in a VM, on your Win2k/XP laptop, while browsing wirelessly == Ads blocked + Internet content cached + More secure.

--Testing (Freebsd / PC-BSD / Nexenta / Solaris / Linux) + ZFS + Samba in a VM when you don't have extra hardware to dedicate to it.

--If you're not already a VM-type person, you wouldn't understand.

Xen "Just Works" (I know. I use it every day)

[eno2001]

Not only that, but I've been running it in a production environment for about a year and I'm about to deploy a HUGE set of servers as VMs using it. Xen beats VMware in one arena: price. If you use the open source version (which I'm doing) it's free. Only VMWare's ESX can compare to Xen. And unlike some people here have been saying, you DON'T need a special processor for Xen unless you plan to virtualize Windows. In my environment, I'm only virtualizing Linux, so I can use regular x86 CPUs dating back to 1998 for Xen. The only exception is the deployment of Zimbra I'm going to do. It requires Redhat Enterprise Linux 4 and NPTL, so I can't run it paravirtualized, it must run HVM which requires the special processors. However, who today isn't getting new hardware with HVM support?

Currently my two Xen servers here at work serve out about four VMs (all paravirtualized on older hardware) for critical and I/O intensive tasks like proxy servers for nearly 1000 machines, or the firewall syslog server for a dual T3 link with about 5000 users behind it sucking the bandwidth dry. So you can't claim it doesn't perform either. Now, if you want point and click administration and an easy set up, then yeah, Xen is behind the times. But performance wise it's leaps and bounds above VMWare. Trust me, I was a VMWare fan before you were in virtualization diapers. And I still am for some applications. But for places where I need something to be cost effective AND give me the features of VMWare ESX, Xen is the ONLY answer.

Lies, all lies

[C_Kode]

Another question hanging over Xen performance concerns the availability of paravirtualized drivers for Windows.

This isn't true completely. The problem is you cannot get these drivers by downloading the OpenSource Xen. You MUST buy the XenSource version. If you run Windows on the *complete* open source version, your network throughput is going to suck like you would not believe. You have to use the XenSource version to get the paravirtualized drivers that bring the network performance closer to what it should be. Virtual Iron has a set of drivers also. (which I believe are better than Xen's, but don't hold me to that)

I found a lot of great insight about virtualizing from Xen to VMWare to Virtual Iron and others on this site. http://ian.blenke.com/xen

Xen is ok

[jshriverWVU]

I've tried both and VMWare is just better. I respect Linux/GPL and the OSS movement, though the main reason I use linux is not just those reasons it's because IT WORKS. So when it comes down to Xen or VMware I use vmware because it works better.

Xen is FOSS so there is potential for them to catch up and with the nature of FOSS new ideas can be tossed in easier. So when that day comes I'll gladly switch over, it's just not there yet.

Xen (and virtualization) is for the Enterprise

[xzvf]

While it might be nice if all these things are easy and work well for the hobby crowd, the real money in virtualization is in the enterprise space. Most servers in enterprise environments run 15% max and are refreshed every 3-5 years. The special processor matters less in that case, and the competition is between a mature VMWareESX server (not free), a hardware based IBM and Xen. Microsoft is a surprisingly minior player. VMWareESX server is very good for x86 consolidation and saves customers money, but is very expensive. It is still the best option for Intel based consolidation. Xen has deep penatration in enterprise lab environments. It is just getting the enterprise management tools to move into real production. IBM is very good at virtualization and stability, but on proprietary power and mainframe hardware. Xen will be fine, because the market is very immature, but expect more seamless and non-attrusive virtualization on the desktop.

Re:Xen (and virtualization) is for the Enterprise

[b0r1s]

A couple notes ....

1. Microsoft is becoming a bigger and bigger player by the day. I know one managed service provider who's pushing virtually every client onto MS Virtual Server whether they ask for it or not.
2. Virtualization is selling a LOT of units in the hosting industry, and as people realize how convenient it can be, it's moving to smaller shops
3. Shit like Virtuozzo / OpenVZ seems to get ignored because it's so hosting centric - there's a hell of a business model to be made in turning something like SW-Soft's VZ management suite into an enterprise product rather than a hosting-centric billing/support control panel.

But what do I know, I just do this for a living.

Re:Xen (and virtualization) is for the Enterprise

[xzvf]

Good points. Hosting companies are leaders in the virtualization market, and most of what I was talking about before was large enterprises consolidating. I have a limited field of vision on this issue, but I just don't see the buzz around the MS offerings with big customers. I think we have some emerging contenters in the virt market, like KVM and VZ and maybe even Sun's containers. What would be great is an overarching enterprise level management tool that takes advantage of the various hypervisors (container, guest) to use the best in the correct situation. We also need improvements in the network infrastructure to make is far more flexible and dynamic when moving VM's around.

Re:Xen (and virtualization) is for the Enterprise

[Dorkmunder]

Yes, Xen is becoming a great tool for hosting companies, making it very easy to offer customers quickly scalable resources (look at www.engineyard.com for an exmaple on Xen on Gentoo with Mongrel and Rails IIRC). It has also become a great resource for me in a University setting where VMWare is just too dang expensive. We run Xen on Debian and then "guest" all sorts of OS'es (yes, we have the right processors to make Win2003 work). On the down side, they still have some work to do to make multiple virtual volumes a possibility plus they need to have a way to migrate guests from server to server across a SAN (right now you need to do an export\import vs. VMWare's VMotion app. Supposedly all of this is coming this year in the next version (3.3? or 4?). Anyway, I have found the performance to be excellent and the cost right.

Re:Xen (and virtualization) is for the Enterprise

[The Second Horseman]

Expensive? Not really, if you compare the costs of actually getting that number of servers. Given the feature set you get, it's pretty modest. If you work in an educational setting, it's even cheaper. You can get VI3 Enterprise and a tier one server - 2U rackmount, dual quad-core system (2.4 ghz) with 16GB of RAM, dual power supply, 6 hour CTR service, an additional NIC and a 4GB fibre channel card and about 512GB of local storage for about $16,000. Depending on the size of the VMs you need to run, you can easily get anywhere from 12 to 20 VMs on that server.

We're entering our fourth year of server virtualization under VMware, and we would've sunk without it - trying to meet customer demands in our space and budget would have been impossible.

Factoring in things like the cost of gigabit ports on a data center switch, cost of power outlets (distribution is often as much of a problem as capacity), and the cost of fibre channel switch ports, there's a huge savings per server. We've got 42 VMs running on three servers similar to the above (one 2xdual core, two 2xquad) and an older server still running VMware ESX 2.5.x slated for replacement next year.

And yes, fibre channel. It's still cheaper than 10GB ethernet, and can run at 4GB.

Re:Xen (and virtualization) is for the Enterprise

[jimmyharris]

One thing that no-one seems to have mentioned is Red Hat's new licensing model for Xen-based RHEL installations.

Red Hat Enterprise Linux Advanced Platform allows you to run an unlimited number of guest RHEL instances without buying any more licenses. That's right, whereas previously you would have had to pay $799/$1299 pa for the host and every guest, now you are entitled to run as many RHEL guests as you like, just for a single $799/$1299 pa cost. (more info)

Assuming you run ten RHEL guest instances per server, that means a saving of $7990/$12990 pa per server.

Not to mention that XEN itself doesn't cost anything extra, unlike VMWare. Having said that, VMWare ESX is certainly in a different class to XEN and has a much better support for enterprise environments. But XEN's improving and this licensing change will show large license cost savings for migrating to XEN.

Re:Xen (and virtualization) is for the Enterprise

[The Second Horseman]

If you're on some sort of agreement - like Novell's MLA/ALA/SLA licenses, there's a good chance that you're not paying per-server, you're paying a set amount for a product suite per person (full time equivalent employee, student, etc). So SuSE Linux Enterprise Server doesn't cost extra as you add virtual servers, whether you're using VMware or Xen.

Guaranteed Results

[catdevnull]

If you want to get a colorful thread of comments started on slashdot, there are 3 ways to do it with guaranteed results:

1) Say something bad about linux (or about Apple).

2) Say something good about Microsoft (or about Apple).

3) Throw a grenade in the room about Open Source software like this:

The vast majority of those articles -- including a few I've written myself -- take it as an article of faith that Xen's paravirtualizing technical approach and open source business model are inherently superior to the closed source alternatives from VMware or Microsoft.

I'm not making any value judgements here--I'm just amused.

Re:Guaranteed Results

[JasterBobaMereel]

Or say the less extreme versions

1) Linux is not all good

2) Microsoft is not all Bad

3) Opensource is not the solution to all problems, closed source can be better

Re:Xen "Just Works" (I know. I use it every day)

[Anonymous Coward]

Currently my two Xen servers here at work serve out about four VMs

So how many really? Can't you tell?

Xen and OS X question

[Nimey]

I'm getting a Macbook soon, and I want to play around with virtual machines on it.

Is it possible to install e.g. Debian as my host OS, apt-get install xen, and then install Mac OS X inside a Xen virtual machine? This computer has a C2D processor, which supports the Intel VT instructions. I'll also do the same with Windows XP and Vista, and Ubuntu.

If it will work, how well? Will it be a transparent install so that X can directly access the 3D acceleration hardware?

Thanks.

Re:Xen and OS X question

[Anonymous Coward]

You had better take a close look at Apple's license. It prohibits running OSX on other than Apple hardware. OSX checks before starting. There are accounts on the net about modified versions (which disable the check) running on recent x86 hardware so there are no real technical reasons you can't do it. It's just that the licensing prohibits it.

Re:Xen and OS X question

[Nimey]

What the hell does that have to do with it? It's a *Macbook*, which is by definition Apple hardware.

Re:Xen and OS X question

[SanityInAnarchy]

It's also a virtual machine.

I remember, when I first got WinXP on this machine (free, courtesy of my school), I installed it in Qemu. I then tried to install it directly on the hardware, and it insisted that it was a different computer, and I would have to re-activate it. I called MS, and I had to explain to a woman with a thick Indian accent that it was the same machine. (She didn't have a clue about virtualization.)

The problem is, legally, is a VM the same computer, or a different one? It's one of those tricky philosophical questions...

In any case, the Vista license answers this specifically -- if I recall, only Ultimate may legally be installed under VMs, and a VM is clearly enough defined, I think, that you can't pretend that a Transmeta processor or the Intel/AMD microcode is at all like a VM.

Re:Xen and OS X question

[Nimey]

OK, fine, but let's ignore the legal question (since it'll be the only copy of OSX running) and go technical.

Will it work?

Re:Xen and OS X question

[Ash-Fox]

You will need to use the Project OSX86 patches to get it running on the majority of virtualization software out there currently.

Keep in mind that this invalidates your license to use the software in more ways than one.

Re:Xen and OS X question

[SanityInAnarchy]

Also note: The OSX86 stuff, if I remember, is based on a fairly old kernel, because at the time, OSX had to run on real PC hardware, which had a real BIOS. Newer kernels run on OpenFirmware machines, just like on PowerPC...

But I don't really know what I'm talking about; maybe I'm entirely wrong.

Re:Xen and OS X question

[Doobian Coedifier]

I recently got a Macbook, and have been using VMWare Fusion, which is currently a free beta. It's worked well with my Fedora 5,6 and CentOS 4,5 VMs, haven't tried Gentoo yet. Pretty polished for a beta, no complaints so far...

Currently using Xen...

[BJZQ8]

My company is currently using Xen on something like 40 "virtual machines" on 6 "real machines." Works almost flawlessly. Runs heavily-used multi-gigabyte MySQL databases and Java web apps without complaining. You can move virtual machines between real machines while they're under load, with a 6ms delay. If a developer wants to try something weird, go ahead. If you hose the system, I'll just re-image it and have you going again in 5 minutes. There's nothing wrong with Xen at all, if it's done right. It's ready for the datacenter, because we use it now.

Re:Currently using Xen...

[Courageous]

You can move virtual machines between real machines while they're under load, with a 6ms delay.

That's really interesting. See, the authors of Xen say live migration isn't ready and that it's unstable. I have deployments of redhat, suse, and open source xen that prove that with swarm-cloud migration testing (just put all the vms into flight constantly, ping-ponging around to various hypervisors). Meanwhile, Xen Enterprise does not today feature live migration. Why? Did I say that it's not ready, and that the authors say that it's not ready? Xen Enterprise is the author's commercial implementation of Xen. What do they know that you do not?

That live migrations fail, occasionally, for no apparent reason. Most common failure: live migration never ends, get stuck with a zombie on other end. Rarer failure: host operating system crashes, taking all virtual machines with it.

Don't get me wrong. I have high hopes for Xen in the future. But advertising Xen as "data center ready" TODAY is irresponsible. It just isn't ready for anything past judicious and careful early deployment.

C//

Re:Currently using Xen...

[BJZQ8]

Well, I've done numerous tests of live migration, and it works for me. Do I know something they don't? I don't think you can say either way. Perhaps on somebody else's network, it might not work. But I've done migration under heavy load and not had any failures, zombies, or crashing of Dom0. I had to clarify the 6ms delay in another post. Here's the way I put it, simplified because I screwed up the formatting...

ping...64 bytes from xxxx...5 ms
ping...64 bytes from xxxx...5 ms
ping...64 bytes from xxxx...11 ms
ping...64 bytes from xxxx...5 ms
ping...64 bytes from xxxx...5 ms

    The bump to 11ms was the transition between real machines.

  We use it because we like to have lots of machines doing different things at different times. The developers have wild ideas, so we give them what they want as far as an OS, and create and destroy as necessary. We keep the migration as an option in case of failure; we don't do it on a regular basis.

Re:Currently using Xen...

[Courageous]

I understand the latency of the switchover. It will be dependent on the size of the volatile set of memory that needs to be transferred between the save/restore cycles. I.e., this will be virtual machine-dependent, and tend to increase linearly with the virtual machine's memory footprint and memory utilization.

"Data center readiness," to me, does not mean a few servers running Xen. It means many, many servers, taken from at least superset of servers taken from all the mainstream enterprise server vendors, in all the common configurations. IOW, if I have a set of Dell 1955 Blades with Net App 3050 NFS storage heads running Xen and it crashes during live migration, (and I do), Xen is not ready for data center deployment, QED.

An example of someone else who's jimmied together a successful rig does not serve as evidence of "data center readiness." I have tried SLES10 out of the box, SLES10+patches, RHEL5, RHEL5+patches, and Open Source Xen with careful compilation of specific kernel options... all with live migration. They are all terribly unstable on our hardware configuration.

Show me an enterprise deployment of hundreds of Xen hypervisors deployed servicing guests with a minimum of four nines of availability guarantee with a contractual financial penalty for availability violations... then we can start talking seriously about the "data center readiness" of Xen.

Today Xen is /not/ "data center ready".

Maybe next year.

Not today.

If you ask me, Xen's going to get clobbered by KVM; a different subject for a different day.

C//

Re:Currently using Xen...

[BJZQ8]

I have used RHEL5 and FC6, and had no stability problems. Obviously your requirements are higher than mine, and your datacenter larger than mine. I salute you.

Re:Currently using Xen...

[pyrrhonist]

Show me an enterprise deployment of hundreds of Xen hypervisors deployed servicing guests with a minimum of four nines of availability guarantee with a contractual financial penalty for availability violations...

Can't promise four nines. Will you take nine fours? Then you can trade fours with someone later.

Re:Xen "Just Works" (I know. I use it every day)

[div_2n]

The only exception is the deployment of Zimbra I'm going to do. It requires Redhat Enterprise Linux 4 and NPTL

Last I checked, Zimbra runs on Ubuntu 6 just fine.

Re:Xen "Just Works" (I know. I use it every day)

[eno2001]

It depends on what I'm doing. If you weren't trying to be cute, I'd say you were trolling. In reality, it's very common practice to use LVM to clone a filesystem, make some changes to the various files that set IP and hostname as well as other unique host settings and bring up alternate "Test" VMs on a Xen box. So some days I might be running three VMs other days eight or ten. It all depends on what I need to do.

As an aside, I forgot to mention that there are NO other products other than VMWare ESX that offer "live migration" of a running VM from one hardware host to another. That's right... you can take a VM that is running with many users actively using it and move it from one physical box to another with only a few milliseconds down time. The users NEVER notice. The free VMWare server can't do that. Micrsoft's Virtual Server can't do that until they have a hypervisor. And there really isn't anything else that can.

Xen's Maturity

[Krondor]

He concludes that the virtualization technology has some maturing to do...

I RTFA and it says very little about the maturity of the actual Xen technology. The article is more a point about several non-related factors;

1.) There is a lack of pretty management interfaces.

True, but these are in the works from Red Hat, Novell, XenSource, and various other ends. Already some of them look pretty promising, but if you are a real admin you don't need them in the first place. There is nothing wrong with using the command line tools to manage your Xen virtual guest environment.

2.) There is a lack of references for companies using Xen.

How does this relate to the viability of the Xen virtualization? Yeah it makes management feel nice and fuzzy that others are using something, but this does not relate to how well the Xen technology performs. I also suspect that like many open source projects, there are many people using it that do not report it. Novell has personally contacted me and my company to ask us to assist in their new paravirtualized Windows drivers initiative and then be a reference for the technology. It seems that at least some companies are moving to address this, at any rate.

3.) There aren't many benchmarks about Xen versus VMWare.

VMWare does not allow benchmarks they do not approve of. It's in that draconian EULA you agreed to by using it.

4.) It's awkward to paravirtualize Windows.

Yes, it is. Novell signed the soul sapping agreement with MS and as such is pushing some paravirtualized drivers for Windows. The article continually talks about woes with Xen on Red Hat. Red Hat didn't sign the agreement and will require some much more intelligent coding to make this happen. It might never happen, so for Windows it's full virtualization with VT (or AMD's equivalent) or bust. Sorry, use SUSE for it or use full virtualization. It's an MS issue not a Xen issue.

5.) MS's new Viridan Virtualization Platform is using paravirtualization as well.

Yep, that should be a testament to the approach versus VMWare. Though it is interesting that VMWare now has a Linux kernel virtualization implementation similar to KVM. It seems VMWare is headed to paravirtualization as well. Obviously Xen did something right.

6.) There is a lot of competition.

True. How again is this relating to Xen as a virtualization technology.

Again, I'm not saying Xen is perfect. It definitely has issues and room to grow. I'm just saying that the article makes little, if any, relevant points to Xen's virtualization technology.

Re:Xen's Maturity

[drsmithy]

True, but these are in the works from Red Hat, Novell, XenSource, and various other ends. Already some of them look pretty promising, but if you are a real admin you don't need them in the first place. There is nothing wrong with using the command line tools to manage your Xen virtual guest environment.

It's not just about pretty pictures, it's about usability. For example, Xen is a bitch to setup with any sort of non-trivial networking environment (eg: multiple vlans, bonded interfaces, etc). You frequently have to write your own scripts to make it work in such situations, but this requires a very good - arguably completely unnecessary - understanding of what's going on behind the scenes.

Now, while this might serve to make some people feel like their e-dick is huge, for those of us that actually have to get work done, and have a perspective of a picture bigger than writing some scripts to do the job someone else should have, it's nothing but a pain in the arse and a waste of time. Firstly, because it requires hours(/days/weeks) of research and work to get the initial configuration going, that has to be scheduled with all the others tasks in the pipeline. Secondly, because it makes all those machines "different" in annoying ways from other machines with ostensibly similar configurations, thus (needlessly) increasing sysadmin overhead. Thirdly, because the requisite for in-depth knowledge extends to everyone who might have to play any sort of role in managing the system, meaning your understaffed IT department is less effective.

That's just one example. There are many more (eg: live migration of VMs) where Xen can sort-of, kinda, emulate the features present in something like VMWare ESX, but you have to spend weeks researching and testing the configuration, instead of spending twenty minutes following the step-by-step instructions in the (*gasp*) documentation, then forgetting it ever happened.

I've never understood why some people - OSS unix geeks in particular - consider the quality of software to be inversely related to how easy it is to use. I can only assume it's because they're worried the Priesthood would be threatened if people only realised that a nontrivial proportion of "sysadmins" are little more than helpdesk staff who just happen to know the right incantations.

I'm sure there are many people using Xen in "enterprise" scenarios. We are, but only because the manpower is largely already spent. If I'd known before I started pursuing Xen what I know now, my advice to myself would be the same to my advice to most people looking into enterprise-level virtualisation - get VMWare ESX. You might think it's expensive, but it's almost a certainty you'll spend more money on Xen in wasted manpower trying to get it up to half the functionality and relability. Just give them the money and get on with improving your environment in ways that will boost productivity, rather than pouring manpower into a black hole of effort polishing up someone else's half-finished product.

Re:Xen's Maturity

[martyros]

There is a lack of pretty management interfaces.

This seems to be one of his main disappointments, in fact -- specifically, that RHEL 5 doesn't have pretty management interfaces. (The only mention of XenSource and Virtual Iron's management interfaces seems to be to re-emphasize that RHEL's GUI is really bad.)

The Xen hypervisor is an engine, not a car. Xen is in some ways similar to where Linux was in the late '90s -- the kernel worked great, but the GUI was way behind. And the fact is that corporate customers need a complete solution, not just a great "engine". The number of cool features of the OSS Xen is really astounding -- but without a convenient way to manage it for mere mortals, it's not that much use to most IT folks. And getting that functionality, along with other functionality that IT folks might need (like say, storage management) into a useful GUI is a very difficult proposition, and not at all related to the virtualization technology itself.

Re:Xen's Maturity

[martyros]

You frequently have to write your own scripts to make it work in such situations, but this requires a very good - arguably completely unnecessary - understanding of what's going on behind the scenes.

You seem to be complaining that the Xen hypervisor doesn't do anything with networking. That's not what a hypervisor is for. Setting up virtual networks is outside the scope of a hypervisor; that's why you need other tools. It would be like complaining 10 years ago that Linux worked fine for process management, but that it was a pain to set up networking, that you needed to edit all kinds of scripts and nonsense. That's got nothing to do with the kernel, but with the lack of supporting tools to make configuration easy. Just like the graphical tools needed to be developed for networking and other configuration for Linux, the tools for management still need to be created for Xen.

VMWare has been developing their management interfaces and infrastructure for 6-7 years now. Don't be surprised that RedHat, XenSource, Virtual Iron, et al don't have parity after less than a year of working on it.

Re:Xen's Maturity

[FireFury03]

For example, Xen is a bitch to setup with any sort of non-trivial networking environment (eg: multiple vlans, bonded interfaces, etc).

I'll agree with this, although it isn't the hypervisor's fault - it's the userland stuff that's at fault. For example, Xen doesn't appear to support IPv6 *at all* in routed mode, I had to hack up my own scripts to do it (and I'm seriously considering moving over to bridged mode in an effort to simplify and standardise my system). But I'm curious - do other virtualisation systems handle these sorts of things any better?

I've never understood why some people - OSS unix geeks in particular - consider the quality of software to be inversely related to how easy it is to use.

I certainly don't. However, experience has taught be that software with shiny GUIs is often inferior in other, more important areas. That's not to say that I think making things hard is a good thing, but maybe the programmers are fixing problems they consider to be more important. Afterall, Free software is usually written out of necessity (i.e. developers implement the features that _they_ want) rather than having some corporate agenda to make it "easy" for "normal people".

Also, in my experience software with GUIs is also often missing a decent commandline interface - GUIs are all very well but once you have learnt to use a commandline interface then it is (a) much faster to use and (b) usable over a low bandwidth network connection (I don't want to try tunnelling an X11 based configurator over GPRS just to tweak a setting on my server).

If I'd known before I started pursuing Xen what I know now, my advice to myself would be the same to my advice to most people looking into enterprise-level virtualisation - get VMWare ESX.

I've not had any significant problems with Xen (other than the above note about IPv6 support) so in my case VMWare would be a waste of money. I guess each person's needs are different, but for me the act of actually going and buying VMWare (and having to keep it up to date) would make it more effort than just using Xen which happens to be already there on recent distros and Just Works out of the box for most situations.

You might think it's expensive, but it's almost a certainty you'll spend more money on Xen in wasted manpower trying to get it up to half the functionality and relability.

I'm not sure where your reliability concerns come from - I've been running a number of Xen VMs across a number of real machines for quite some time and I've never had one break on me. Also, as mentioned, for what I'm using virtualisation for, I would be expending more man-hours administering VMware than I am administering Xen so I think your arguement is possibly only applicable in some very specific situations.

Re:Xen's Maturity

[Anonymous Coward]

I recently attended a Novell-sponsored seminar in which they talked about the future of their product portfolio. The way Xen was presented there indicated to me that they plan to use it to paravirtualize NetWare for those legacy applications that just can't (or won't) run on Linux. Novell is moving their customer base away from the NetWare base to a Linux (SuSE) base, but they care significantly about backwards-compatibility.

Actually, let me say that again. Novell is moving their customers from NetWare to Linux, with seamless backwards-compatibility.

They didn't have to do that, nor did they have to worry about backwards-compatibility. But if a customer has to replace applications to change platforms, they might even switch to Windows. I can't imagine Novell would want that to happen. And using Xen, they're doing it right, instead of trying to bolt NetWare's micro-kernel into Linux (like Vista and legacy code).

It never struck me that Novell considered it quite as important for Xen to virtualize anything as much as NetWare.

-M

Re:Xen's Maturity

[turbidostato]

"Also, in my experience software with GUIs is also often missing a decent commandline interface - GUIs are all very well but once you have learnt to use a commandline interface then it is (a) much faster to use and (b) usable over a low bandwidth network connection"

Even this is of lessen importance to me. To me, the *real* problem with GUIs is that they lack three most important characteristics for any corporate environment: traceability, repeatability and (arguabily related) self-documentation. I can take any day, say /etc/hosts, and immediatly see what changed from yesterday's situation and usually, who and why did the change. Regarding scripts, once they are polished, I know I can reach exactly the same results one time or one hundred. You just can get this with point-n-click interfaces.

Re:Xen's Maturity

[richlv]

i agree that gui might be important in some cases, but i am holding back until xen will not require patches to vanilla kernel.

what additionally seems weird, the development isn't as open as one would hope for such a popular project.
getting some development snapshot ? any links from the main page ? nooo.
in the end i managed to grab a copy from versioning system they used, but it was several versions behind vanilla kernels (that had some nfs patches that i needed), and i didn't feel like dissecting the patches or using older kernels only to play around with xen.

getting in the mainstream kernel would mean closer development cycle to the kernel itself. which is usually a good idea for anything dealing with the kernel.

Re:Xen's Maturity

[Raenex]

You just can get this with point-n-click interfaces. Sure you can. You just have to build it into the GUI. The GUI can generate logs, scripts, whatever. Ideally all software would have compatible GUI and command line interfaces.

Re:Xen "Just Works" (I know. I use it every day)

[Courageous]

The free VMWare server can't do that. Micrsoft's Virtual Server can't do that until they have a hypervisor. And there really isn't anything else that can.

Well you can try to do that with Xen if you want, but you might be sorry. :) But I suspect you knew that.

Hopefully the Summer release remedies this situation.

C//

Xen vs. VMware

[TheRealFixer]

I'm primarily a VMware VI3 user, but I've been starting to do more with Xen lately. I have to say, Xen is very impressive in what it accomplishes. It's very stable, and has the capability to do some really advanced stuff. That being said, it can be a real pain to get some of those advanced features working. For example, running Xen in CentOS 5, I had a server with two NICs, and I wanted to setup a second bridged interface for the second NIC. It took way more effort than it should have to get that working correctly. In ANY VMware product, that's a task that takes, literally, seconds. Another thing I'm working on is getting VMs to auto-boot in a particular order, and wait on another VM to finish booting before the next one starts. Again, a task that's second nature in VMware, but appears to be difficult to implement in Xen.

In the end, it all boils down to management tools. There are no decent centralized tools to manage a farm of Xen servers right now, let alone manage just one host. Virt-manager is very helpful, but extremely limited. And I've looked into some web-based management applications, but none of them are anywhere close to mature enough to deploy in a data center. Xen is still my choice for free Linux virtualization, but they've got a long way to go to even approach VMware.

Xen Management Apps Are Not Good.

[tji]

I have been trying to use Xen at home to test it out and compare it to VMWare, which I've used at work. Once you manage to get Xen clients working, it's fine. It does a good job of running VMs, and can be used to partition resources on a powerful machine.

But, the main problem is the steep learning curve for getting Xen running in the first place. The (python based) management GUIs included with Fedora or Ubuntu are weak at best (although, slowly improving.. the UI in Fedora 7 does manage to make setup easier than the command-line alternative). The ongoing management / monitoring of VMs is okay, but weak in comparison to VMWare.

There are also a lot of little quirks in Xen. Installing Win2k in a client VM required a lot of searching for how to attach an ISO image to a running VM (it's not a simple GUI operation like in VMWare/Parallels/VirtualPC, it requires a terminal command with unintuitive options, which never worked for me.. I finally dug out my CD and got the physical CD drive to attach to the VM). Windows VMs have an odd issue where the mouse pointer is offset form the actual pointer (it's a known issue, and is helped by turning off mouse acceleration in Windows preferences, but it is still a problem). Installing client VMs can be challenging.. Ubuntu feisty wouldn't install until I set the VM as a Solaris client, and after a few other tweaks it finally installed and worked fairly well.

Most of the Xen problems are solvable, after playing with command-line tools, figuring out poorly documented parameters, and lots of googling. At the end of the day, it's one of those "Xen is free, if your time has no value" type things. VMWare Server is probably a better option if you just want it to work for home/free uses. For commercial use, VMWare ESX Server is the way to go. It has simple VM setup for many client OS's, excellent management of large groups os Hypervisors and virtual machines.

The commercial alternative from XenSource (free to use, but limited to 4 VMs; or less restricted versions for increasing $$) offer a better management UI, but are too restricted for my taste. The management app is much better, but not as good as VMWare.. If I'm going to pay for one, I'll go for the best option.

Re:Xen Management Apps Are Not Good.

[Blackknight]

Your mouse issues can be fixed by setting the VM to use USB tablet emulation instead of mouse. This is documented on the mailing list if you run a search.

Put this in your config file.

ne2000 = 1
usb = 1
usbdevice = 'tablet'

Using an ISO is easy as well.

For example, here's my config for using the Longhorn beta iso.

disk = [ 'file:/xen/images/longhorntest.img,ioemu:hda,w', 'file:/xen/wsl_6001.16510.070417-1740_x86fre_serve r-KB3SFRE_EN_DVD.iso,hdc:cdrom,r' ]

I agree that the GUI management tools aren't the greatest, it's a lot easier to just use the command line. Real server admins don't need GUIs any way.

Re:Xen Management Apps Are Not Good.

[tji]

Thanks for the info. I'll try the mouse change. But, these solutions are a good illustration of the difference between VMWare and Xen.. In VMWare, these things either work without modification, or are easily done via the GUI.

The core of Xen is great, and it will be fantastic if some good GUI management tools are available. But, in general, open source and easy to use GUIs don't often go together.

Also, on the disk issue, that config file change works for booting from the CD. But, after I reboot without the CD, and boot from the hard drive re-attaching an ISO is not so easy. I think the command was "xm attach ...." and I never got the ISO to work, but pointing it to my CD-ROM device did work. Removing attached disks was also a challenge. Not rocket science, but not something I want to mess with either.

Re:Xen "Just Works" (I know. I use it every day)

[tji]

"Just Works" is overstating it quite a bit. Based on my experience, and looking at the other comments here, it's more like Xen "mostly works, after a great deal of learning, googling, and experimenting". Maybe once you've ramped up on it it works well. But, saying it "Just Works" is clearly not the case.

The Xen experience has improved a lot. In Fedora 7, I just had to select the Xen kernel+apps for a package install, and the Xen infrastructure was pretty easily installed. But, getting client VMs running, figuring out the command-line tools and their parameters, and working around quirks too a good deal of effort.

Summary of article:

[Larry_Dillon]

Not enough market share and immature (optional) GUI == Not Ready for Prime Time.

"Oh my. Editable XML configuration files, obscure command line interfaces, grayed out options in the GUI? Thanks, but no thanks. This thing doesn't sound like it's ready for prime time in Data Center USA."

I say if you can't use the command line YOU'RE not ready for "prime time in Data Center USA."

Data Center USA

[fyoder]

I stopped reading the article with this quote:

Oh my. Editable XML configuration files, obscure command line interfaces, grayed out options in the GUI? Thanks, but no thanks. This thing doesn't sound like it's ready for prime time in Data Center USA.

Are sysadmins at "Data Center USA" morons? "Oh nooo, command line time, I hate that. Oh nooo, my option I want is all grayed out! Help me, help me! Oh I am so sad now."

Deploying vm stuff is not the same as using a word processor. "Data Center USA" is in real trouble if their sysadmins aren't any smarter than regular desktop users.

Re:Data Center USA

[segedunum]

Are sysadmins at "Data Center USA" morons? "Oh nooo, command line time, I hate that. Oh nooo, my option I want is all grayed out! Help me, help me! Oh I am so sad now."

I'm afraid that's about your average ignoramous comment modded up to +5 Insightful on Slashdot.

Such tools that people want to rely on, oh I don't know, all the time, require good management tools in order to set things up in a straightforward way that can be documented and can be reproduced, and that sys admins can use to find out what on Earth is actually going on in their data centres. Sys admins are many things, but telepathy they don't do very well.

Do you really think editing a bloody XML file is a good idea?

Re:Data Center USA

[jimicus]

The whole point of virtualisation products is to make running a bunch of servers easier.

If the virtualisation product can't provide a half-decent API for scripting and a UI for day to day use, then it's running behind VMWare, which can do that. And at the end of the day, if "being able to do everything on the command line" was considered so vital to businesses, there would be no such thing as Windows Server 2003.

Re:Data Center USA

[amchugh]

Command line is fine, but the implication that this is a moving target for scripting isn't.

Re:Data Center USA

[turbidostato]

"Such tools that people want to rely on, oh I don't know, all the time, require good management tools in order to set things up in a straightforward way that can be documented and can be reproduced"

Like, uh... a script? I had always problems trying to understand the rationale of "documented an reproductible" and "GUI" in the same sentence. Can you really talk about "can be documented and can be reproduced" on bold face when all you have is a doc document an some screen captures? Can you really talk about "can be documented and can be reproduced" when something suddenly starts to fail and you don't have the slightest idea about what was changed from yesterday?

"Do you really think editing a bloody XML file is a good idea?"

Not at all. But the ability to diff it to yesterday's version to see what changed it is. As it is running an script from cron instead of waiting for somebody to click on a GUI's box. As it is to have an scripted deployment environment you can guarantee repeatability with instead of depending on the read abilities of a sysop going through a doc with screen captures.

Re:Data Center USA

[Bazouel]

How about the sysadmin softwares get as smart as (some of) their regular desktop cousins ?

If the GUI is stupid or just plain bugged, what does it say about the back-end ?

Sysadmins != Masochists.

Re:Data Center USA

[fyoder]

Sysadmins != Masochists

:) Are you sure masochism isn't part of the job description?

I like easy as much as the next non-masochist, and if the author had said xen wasn't ready for the desktop, that would be one thing, but by his standards it would seem there's a lot of things in data centers that aren't ready for the data center.

sysadmins rock. Remember to express your appreciation on sysadmin appreciation day.

Xen corrupts data

[pseudorand]

I was a big fan of Xen until I found out it was silently corrupting data. I've only duplicated the problem on an older 3ware card, but I was just about to trash the card, drives, etc. when I discovered that I couldn't duplicate the problem when using a non-xen kernel. Now I'm just waiting for pacifica and ivt solutions to mature before I touch virtualization for anything important.

Re:Xen corrupts data

[Anonymous Coward]

My experience with Xen has been sketchy too; I stopped using it after 3.0.2. VMs that don't die, failed VM state saves, failed migrations, the list goes on.

OK, maybe I didn't buy the commercial version, but it's a bit discouraging when the free download page http://xensource.com/download/index_oss.html has the text:

"The downloads on this page have been built from the latest Xen 3.1 official release. They have undergone limited testing, are not supported in any way whatsoever, and will be fixed on a best-effort basis only."

Even if I *was* using the commercial version, this doesn't inspire much confidence.

6ms???

[pem]

Umm, assuming zero overhead, even with 10Gb ethernet, you could only move around 7.1 MB in 6 milliseconds.

With Gigabit Ethernet, you can divide that down to well under a megabyte.

While I'm sure your virtual machines are nice toys, they ARE just toys.

Re:6ms???

[msh104]

naturally he is telling you that there is an added 6ms to the response time on your server WHILE it is transfering.

Re:6ms???

[pem]

But, that would be a highly application dependent, highly speed of transfer of VM dependent, highly CPU and disk speed dependent, completely useless statistic.

You're right, never mind, just remembered I'm on Slashdot.

Re:6ms???

[MajinBlayze]

YMMV, depending on usage during the time of the switch, but Xen starts migrations by copying over memory *while the original VM is running*. Then, the original VM is suspended, checked one last time for data consistancy (the delay), then the VM is brought back up by the new host.

[PDF warning] Live Migration of Virtual Machines

Re:6ms???

[BJZQ8]

Maybe I should clarify some things, since I'm on Slashdot and can't expect people to take things in context. You are pinging an active machine that is to be moved. Here's what it looks like: 64 bytes from xxx.xxx.xxx.xxx: icmp_seq=1 ttl=64 time=5.01 ms 64 bytes from xxx.xxx.xxx.xxx: icmp_seq=2 ttl=64 time=5.21 ms 64 bytes from xxx.xxx.xxx.xxx: icmp_seq=3 ttl=64 time=5.10 ms 64 bytes from xxx.xxx.xxx.xxx: icmp_seq=4 ttl=64 time=11.10 ms 64 bytes from xxx.xxx.xxx.xxx: icmp_seq=5 ttl=64 time=5.01 ms 64 bytes from xxx.xxx.xxx.xxx: icmp_seq=6 ttl=64 time=5.31 ms That bump in latency was the transition. You have officially gone from one machine to another. That's what I meant by 6ms. I have tested this over a range of CPU usages, and it's pretty consistent, at least on my network (full gigabit Extreme switches, newer Dell 4-CPU servers, SAS drives, lots of RAM.)

Re:6ms???

[MajinBlayze]

Another thing that is important to note, according to the page I posted (seriously, it's a good read) is that downtimes, depending on IO load can be between 60 (Quake 3 server migration) to 210 (SPECweb benchmark utility) ms.

You will also find in the link information on a test where 6 players were connected to (and actively participating in) a quake 3 server. It was migrated across machines without dropping a single player.

Parallels

[queenb**ch]

Parallels, which is a commercial product, is one of the slickest VM's I've seen for desktop use. No one ever mentions them, either.

2 cents,

QueenB.

Xen kernels are nearly useless

[r00t]

Again, virtualization is for running Windows. Not Linux!

Yeah, sure, there are a few weirdos out there.

For most of us, there is no point in running Linux under Xen. We already gave Linux the native hardware. I guess somebody might want to run a Linux guest on Windows, but that'd be Wrong and is anyway unsupported.

When I want to run a Linux app, I just run it. No problem. When I want to run a Windows app, I need virtualization.

Xen is thus a solution in search of a problem.

Re:Xen kernels are nearly useless

[Anarke_Incarnate]

Of course!! Why would somebody want to buy big hardware like a Sunfire X4600 M2 with 16 cores, 256GB RAM, 4GbE and multiple I/O slots to run multiple instances of Linux and/or Windows and/or BSD, and/or Solaris x86 when they could just run 1 instance and let the computing power of their server be use damn inefficiently?

Nobody wants to make their data center streamlined and efficient for use of power and cooling, which in many ways costs more than the initial hardware purchase over a 4yr refresh cycle....

/sarcasm

Re:Xen kernels are nearly useless

[stevey]

That might be true for you, but not for other people.

I run Debian unstable upon my desktop, but if I want to build Debian packages suitable our the Stable release, or even CentOS5 RPMs and SuSE RPMs, what should I do? I could use chroot(), or I could use virtualisation. That latter is what I do, because it is more "real" and allows me to run X, etc.

There are many times when it is useful to run distinct copies of Linux upon one host, for example a hosting company giving user's their own "real" system, to build packages for multiple distributions, to test software without messing up your development system, automated testing systems, and more.

Yes you can do that without virtualisation, but sometimes it isn't as neat.

Re:Xen kernels are nearly useless

[DrSkwid]

There are more OSes than Windows & Linux

Re:Xen kernels are nearly useless

[r00t]

Why would somebody want to buy big hardware like a Sunfire X4600 M2 with 16 cores, 256GB RAM, 4GbE and multiple I/O slots to run multiple instances of Linux and/or Windows and/or BSD, and/or Solaris x86 Indeed, why would they?

I guess there is some pride in having an expensive single point of failure that is hard to find parts for. Wait...

Pay attention now. This is not how Google runs their datacenter. Unless you are stuck with something like a database that can't cluster, or one that won't cluster because you didn't pay the extra license costs, this kind of hardware is just stupid.

Shop around. One can find power-efficient 1U boxes. Sometimes non-rackmount is better, including weird stuff like the Mac Mini. Be willing to look beyond Intel and AMD. VIA makes some low-power chips.

Re:Xen kernels are nearly useless

[r00t]

Replace "Windows" with your choice of obscure OS as desired. You probably won't be getting Xen support in OS/2, OpenServer, NetWare, etc.

Re:Xen kernels are nearly useless

[drsmithy]

Indeed, why would they?

Because being able to dedicate a "machine" to each service rather than trying to run dozens of different services on the same machine vastly simplifies operations.

Pay attention now. This is not how Google runs their datacenter.

How Google runs their datacentre is not relevant to most people, who have vastly different requirements, budgets and capabilities.

Shop around. One can find power-efficient 1U boxes. Sometimes non-rackmount is better, including weird stuff like the Mac Mini. Be willing to look beyond Intel and AMD. VIA makes some low-power chips.

Indeed. Then instead of the 2-4 power connections, ethernet connections, fibre channel connections, the cooling capabilities, electrical capacity, 4RU (or less, with other hardware options) of rack space, etc you need for a single machine running 30 VMs, you need *60* power/ethernet/FC ports, higher cooling and power needs, 30 rack units, 30 KVM/serial/RJ45 ports, etc.

Price those out in a datacentre and suddenly that "expensive, single point of failure" becomes cheap and easy to turn into 3 or 4 machines running VMs.

That's just the basic physical footprints - this is before even getting into the _manageability_ advantages of VMs over physical machines (eg: being able to roll out new servers by running an install script and coming back 20 minutes later, instead of having to get physical hardware specced, ordered and installed).

Re:Xen kernels are nearly useless

[tinkertim]

Again, virtualization is for running Windows. Not Linux!

And vms, and minix, and amoeba, and a whole bunch of other stuff that doesn't jive up well with a Linux kernel due to its design, or their's depending on how you see things.

I don't *know* why people shoot themselves in the foot fully virtualizing Linux with a Xen HV. Paravirt ops are a proven major happy place which is why they're in the upstream kernel.

I like being able to make a bios out of thin air, I think its really neat, but why the &(@* would I want to do that with 2.6 under Xen? Special cases being stuff like Mosix 2.6, file systems that don't exist in the sparse tree xen just recently moved away from, other plenty of good reasons.. but this HVM craze has me just chuckling quietly.

Re:Xen kernels are nearly useless

[Wdomburg]

Even ignoring production, virtualization is great for test environments. My desktop is currently pretending to be a four node cluster and an iSCSI SAN.

Two words: OpenVZ

[pyite69]

If you are running the same OS on each VM on a server, OpenVZ is the best.

Performance is great, good control over resources (with the glaring exception of disk IO operations, which they are working on).

don't know v. Xen, but generally

[colourmyeyes]

VirtualBox is really nice; I don't know about the performance comparison to Xen, but VirutalBox is a breeze to use and performance is generally pretty good. I've been messing with VM software for about six or seven years and just discovered VirtualBox a few months ago. Before that I had heard of Bochs, Plex86, Xen, VMware, VirtualPC, Parallels, and Qemu (off the top of my head). Where has VirtualBox been hiding?

Scientology and Mental Health

[Anonymous Coward]

For a moment there I thought the submitter mean 'Desperately Seeking Xenu'.

Doesn't support power management

[Fry-kun]

The only reason so far that's stopping me from using Xen is that it doesn't support frequency scaling and other power management of my laptop.
Sticking with VMWare Server for now...

Re:Xen "Just Works" (I know. I use it every day)

[Phrogger]

> As an aside, I forgot to mention that there are NO other products other than VMWare ESX that offer "live migration" of a running VM from one hardware host to another. That's right... you can take a VM that is running with many users actively using it and move it from one physical box to another with only a few milliseconds down time. The users NEVER notice. The free VMWare server can't do that. Micrsoft's Virtual Server can't do that until they have a hypervisor. And there really isn't anything else that can.

    Hmm, the Virtuozzo servers I administer actually do that rather well. Virtuozzo has had live migration capability (and dynamic resource allocation) for quite a while now.

Xen rocks: In production

[Tracy Reed]

Xen saved my former employer a bunch of money and gained then great flexibility and reliability. They use an AoE (ATA over ethernet) SAN so the compute nodes are totally diskless and all of the data and root filesystems are on the SAN. Now they have email, database, web serving, nearly all of their critical functions in a highly available xen-aoe cluster. I am working with them to release all of the codes and configs in production and we are setting up a website at xenaoe.org (not up yet, but soon) to host the project.

Here is something I wrote up about this architecture for the company when the project went live:

What is Xen?

Xen is a free virtualization system similar to VMware but different. It allows us to run multiple servers/operating systems all on one physical piece of hardware while providing isolation between them.

What is AoE?

AoE is a SAN technology. Similar to Fibrechannel (but far less expensive) or iSCSI (but far simpler and more efficient).

What are the advantages of Xen and AoE for our company?

Xen allows us to more efficiently utilize our hardware resources. The majority of cpu power on your average computer goes unused. Even on servers. They just sit there waiting for something to happen. Even if we get a web request every second the time between one request and the next is an eternity for a cpu running at 2 gigahertz. But powerful cpu's are needed for those short bursts of activity. By using Xen to run multiple servers in their own domains (areas of memory) completely isolated from each other on the same physical hardware we can squeeze more utilization out of our existing CPU's/servers. This means we can get by with fewer CPU's, less rackspace, use less power, and require less air conditioning. By encapsulating the servers into this sort of infrastructure it also allows enhanced management capabilities by allowing the administrator (such as myself) to be able to get console access on the server or restart the server while remote instead of having to drive to the datacenter (which in our case is a 30 minute drive down to Kearny Mesa).

AoE allows us to put a bunch of disk in relatively inexpensive and low CPU powered servers on the network and allow the rest of the servers to access it exactly as if the disk were locally installed in that server. This is advantageous because we can now aggregate all of our disk into one system and treat it like a pool of storage where we can dole out an appropriate amount of disk to each server (often only 10 or 20G is needed) instead of having to put in a dedicated 250G disk which is the minimum you can easily buy these days and waste a lot of disk and power to run it.

The combination of Xen and AoE allows us all of the above plus some interesting fault tolerance abilities. There are now two levels of redundancy in our disk systems and an extra level of redundancy in the cpu's also in that if one cpu fails (or the associated motherboard, RAM, or network card) we can easily switch the servers that were hosted on that machine over to another cpu on the network with either zero or very minimal downtime whereas previously that kind of failure would have required me to drive down to the datacenter and shuffle hardware around or buy new hardware to replace the failed system which all takes time and can result in prolonged downtime.

Re:Xen rocks: In production

[Anonymous Coward]

You could stand to shorten your sentences a bit.

Re:Xen rocks: In production

[charlesnw]

I have some questions about your post.

What is AoE?

AoE is a SAN technology. Similar to Fibrechannel (but far less expensive) or iSCSI (but far simpler and more efficient). How is AoE simpler then iSCSI? I have deployed both Fiber Channel (very difficult and a major pain) and iSCSIS (which takes about 5 minutes. Plug in ethernet cable, configure interface, install iscsi tools, mount storage). I agree that Aoe is most likely more efficent as its closer to layer one.

having to put in a dedicated 250G disk which is the minimum you can easily buy these days and waste a lot of disk and power to run it. Um. I don't agree that 250G disk is the minimum you can easily buy. Both Dell and HP sell smaller versions (36 and 72 gb).

Other then that I think its a good writeup.

Re:Xen "Just Works" (I know. I use it every day)

[jimicus]

As an aside, I forgot to mention that there are NO other products other than VMWare ESX that offer "live migration" of a running VM from one hardware host to another.

My understanding is that for that to work, you need both the source and destination host server to have access to the same physical disk - either through shared SCSI, fibre channel, iSCSI or similar.

I've looked into the price of VMWare ESX (or VMWare Infrastructure as they now call it). As far as I could gather, if you've got the money for a half-decent small SAN for such live migration, the price of a couple of ESX licenses isn't a huge issue.

Or am I wrong?

Re:Xen "Just Works" (I know. I use it every day)

[Anonymous Coward]

Yes, you're wrong. You can set up iSCSI over Gb ethernet using nothing more than a normal Linux box with normal disks.

You can even have a pair of servers with a GigE crossover running NBD to provide 'network RAID 1' of the disks with transparent hot failover.

Re:Xen "Just Works" (I know. I use it every day)

[TooMuchToDo]

You're very right about VMWare ESX. We use it in production for a couple thousand users, and I'm still in awe that I can push running VMs from one physical box to another with less than a second of downtime.

Re:Xen "Just Works" (I know. I use it every day)

[eno2001]

You can also use global network block devices with a Linux box as your storage server. In my case here at work we've got a SAN, but we als have budget cuts and we're a non-profit... So I can't afford ESX. At home, well... I just like having enterprise functionality without the cost. :)

Re:Xen "Just Works" (I know. I use it every day)

[WuphonsReach]

ESX pricing is in the multiple-thousands of dollars per machine. Which, if you're putting together a collection of $20k-$30k machines probably isn't that bad.

But it's horribly overpriced for the smaller market. Picture a small company with 4-12 servers in the $3k-$5k range and a $10k SAN unit. They'd like to be able to pool their servers so that if one box goes down due to hardware failure, services continue to be available.

VMWare's pricing makes that a non-starter.

Xen Enterprise pricing is a lot more reasonable for a small/medium business. VMWare only seems interested in the "big iron" shops who have budgets of $500k+ per year to spend on software licenses.

Hell, in a small business, you don't even care if there's a few minutes of downtime. At least, ss long as you can get things back up and running quickly on one of the less-utilized servers.

Re:Xen "Just Works" (I know. I use it every day)

[Doug Neal]

I agree. It doesn't "just work" by any stretch of the imagination. When it does work, it's great, but there's a whole mess of shell scripts working in the background which don't handle error conditions very well and you often get presented with very cryptic and often quite misleading error messages ("Backend scripts not working" is one of my favorites).

Re:Xen "Just Works" (I know. I use it every day)

[Anonymous Coward]

Quote:
>As an aside, I forgot to mention that there are NO other products other than VMWare ESX
> that offer "live migration" of a running VM from one hardware host to another.

Xen does this. I've tested it repeatedly using a Dell blade server enclosure with 1855 and 1955 blades utilizing Qlogic/McData switches and a 3Par Data SAN for shared storage.

Xen USUALLY "Just Works"

[Anonymous Coward]

Less than a year ago, when I still worked at Amazon.com, Ulrich Drepper from RedHat came and gave a talk. When he heard that we were using Xen in production, he basically said "you guys are crazy, Xen is not stable enough for prime time yet."

Yes it "just works" most of the time. Don't get me wrong -- most of our experiences with it were good. But I was recently talking to a friend who still works there, and they've narrowed down some really hairy bugs that ended up being bugs in Xen. (They know. They have Linux Kernel engineers who are really good at this stuff.)

Re:Xen "Just Works" (I know. I use it every day)

[mtvsucks]

One of the frustrating things about all of the virtualization stuff is how quickly the products are moving and what feature sets are gained and lost but what products. What was true a few months ago about live migration isn't true anymore.

ESX can do live migration from one physical server to another provided the virtual machine image lives on a shared storage.
Xen can do live migration on a paravirtualized image from one physical server to another provided the image lives on shared storage.
Xen cannot do live migration on a fully virtualized server from one machine to another.
KVM doesn't specify full or para virtualization on their migration page, but it does some sort of live migration, and I'll bet it needs shared storage.

Re:Xen "Just Works" (I know. I use it every day)

[qnetter]

Xen 3.1 supports live migration of hardware-assist VMs just fine.

XenSource's next product release, in beta in a couple of weeks and shipping this summer, includes this.

Re:Xen "Just Works" (I know. I use it every day)

[jimicus]

I was quoted around £12-1400 per machine. Which, considering it allows me to multiply the use I'll get out of a £2500 server by a factor of about 8 or 10, seems to me a good deal.

Though right now I'm using the free server product because I don't have the budget for the SAN (or even to dedicate the tin to a Linux box supporting iSCSI), and without that I don't see the benefit in the expensive product.

I've not seen standalone SAN units at £5000 in the UK - more like £7-10K. But then, US$ price conversion has never been a strong point with most IT suppliers.

It's All About Device Drivers

[trenobus]

Paravirtualization is a way to leverage device drivers in the host OS. Device drivers are usually written for Windows first, and sometimes, for Windows only. As long as this remains the case, Microsoft can take as long as it needs to produce a Windows-hosted paravirtualization that will compete effectively (and unfairly) with Xen. The hypervisor must ultimately control the I/O devices, one way or the other. Even with some future hardware support for virtualized I/O devices, the hypervisor must at least manage which guest OS gets which device, even if the device driver resides in the guest.

The move toward virtualization presents a once-in-a-lifetime opportunity to break Microsoft's grip on the OS. The way to do that is to create open standards for hypervisor interfaces, particularly for embedding device drivers in the hypervisor. Politically, the stakeholders who need to be involved, but do not appear to be, are the PC manufacturers. They have the clout it will take to get device manufacturers to produce device drivers for a new open standard for hypervisor device drivers. What they stand to gain is the ability to at least free themselves from Microsoft's grip on the desktop, and perhaps also to encourage competition in hypervisor products, so that Microsoft is not simply replaced by another monopoly. Just as these manufacturers have some choice in BIOS implementations, they could also have a choice of hypervisor implementations. This would provide many more opportunities for product differentiation.

I call on vendors of both open and closed source virtualization products, and PC manufacturers to start making some open standards now. Otherwise you are going to end up with Microsoft calling the shots - again.

I built a XEN system this week.

[Deadplant]

SuperMicro 3U rackmount chassis with 15 hotswap sata bays
Intel core 2 duo @ 2.4ghz E6600
4x500GB drives.

Ubuntu Feisty Server edition for the dom0 (the root/master OS)
Guest operating systems:
- Ubuntu Feisty server edition
- Windows server 2003 standard edition
- more to be added later.

The disk drive i'm giving to windows is a re-sizable LVM2 partition on a software raid 5 array managed by the root domain.

Less than impressed with Xen.

[Anonymous Coward]

I'd set up a booting, working hard drive and just wanted Xen to be able to run the OS right off the hard drive.

Even asked about it on the mailing list.

Seems that such a configuration is not supported. *shrug*

Linux has inbuilt virtualisation ....

[IncandescentFlame]

I'm little surprised no one has mentioned KVM, the Kernel-based Virtual Machine. Its been included in the Linux kernel since 2.6.20 I think.

Its a module which promotes the Linux kernel to a hypervisor, allowing guest VMs to be run. I have Windows XP running as a guest on my Ubuntu machine at home.

It doesn't present my GTS 8800 as a 3D card to the guest OS though (although neither does Xen). It woule be good to know if there is a system which allows a better hardware representation to be presented to the guest OS.

Re:Linux has inbuilt virtualisation ....

[sglines]

Thanks for pointing to a virtually empty wikipedia article.

Re:Xen "Just Works" (I know. I use it every day)

[mudshark]

I've got Zimbra running in production on Ubuntu Server (edgy) at a sizable school. Not too difficult and I don't have to go anywhere near rpm dependency hell.

Re:Xen "Just Works" (I know. I use it every day)

[namtro]

In my environment, I'm only virtualizing Linux, so I can use regular x86 CPUs dating back to 1998 for Xen. The only exception is the deployment of Zimbra I'm going to do. It requires Redhat Enterprise Linux 4 and NPTL, so I can't run it paravirtualized, it must run HVM which requires the special processors. However, who today isn't getting new hardware with HVM support?

I just recently embarked to get Zimbra 4.5.* installed on a CentOS 4.5 virtual server. It is indeed possible. Below are my (crude) notes.

zimbra install on centos:

1. be sure using NPTL
2. be sure the following are installed
   • yum install sudo curl libidn fetchmail gmp compat-libstdc++-296 compat-libstdc++-33 vixie-cron file
3. disable exim which is pulled in by fetchmail automatically
   • chkconfig --level 2345 exim off
4. run the zcs installer and answer Y for most everything
5. after installing zimbra, comment out the "Defaults requiretty" in /etc/sudoers to get the queue listings to show up in the admin web page (and stop the error)
   • visudo
6. be sure crond service is running...
   • /etc/init.d/crond start
   • chkconfig --add crond
   • chkconfig --list crond

Xen's Pros & Cons

[stry_cat]

I've only been using Xen since RHEL 5 came out. So I'm not an expert. Here are my thoughts on Xen: 1) Hardware cost savings isn't a good as you'd think. For every virtual box, you still need CPU(s), RAM, and harddrive space. i.e. If you've only got X CPUs on in a physical box, you're limited to X-1 virtual boxes assuming just one CPU per virtual box (performance degrades horribly if you start assigning more CPUs than you physically have). The real savings is probably in power consumption, but that only matters if your Google or a Fortune100 company. 2) Don't try to use a GUI on a virtual Linux install. The mouse is horribly broken. RH even admits this in their documentation. 3) The only place to get help is a high traffic email list where your question can easily be lost. Don't even think of asking in the Xen forum unless you're paying Xensource $$$ 4) How-To's are few and far between and often contradictory. 5) WinXP runs extremely well under Xen. I've got 4 in production now. Linux also works fine (I've got 3 in production) as long as you don't need the GUI. 5) #2-#4 will probably work themselves out in the next year or two. 6) Overall in spite of the above problems, Xen is extremely cool and can save your company some resources and make it more productive. It will also only get better as time goes on. If you're not using it, you should at least start to consider it and start testing with some non-critical systems.

done it

[r00t]

Last year we installed many dozens of 1U boxes. At the time, low-power Xeon chips were the best deal. (considering space, performance, and especially heat output) The VIA chips were close.

A big box with virtual machines was completely hopeless in the competition.

Re:done it

[drsmithy]

A big box with virtual machines was completely hopeless in the competition.

It's hard to see how, unless there are some requirements or location specific features you're not mentioning (eg: free rack space/power/cooling). Even at a pretty generous $750 per box for some no-name POS with a single PSU, one SATA drive and a Core 2 Duo CPU, for 40 machines you're looking at $30k just for the servers. But then you also need at least 40 switchports, 40 power connections and a whole rack to put them in. Prices for that vary from place to place, but they're non-trivial no matter where you go.

Now consider a Dell 1955 BladeCentre with three 8-core, 16G RAM blades, plus an AX150i disk array. Cost would be ca. US$35k (probably less if you bought it all at once) and would handle 40 - 50 VMs of the kind you are talking about. However, it would take up 1/4 the rack space (10U), 2 - 4 switchports, at most 4 power connections and spend the electricity and heat budget far more efficiently. On top of that you get the advantages of a BladeCentre and RAID disk array - remote power management, built-in KVM, advanced system monitoring, redundant power, etc. Finally, there's still room for 7 more blades, more than tripling the original capacity in that same physical footprint - something that would require 2 more entire racks to do with 1U servers.

It's difficult to see why anyone would be buying 1U boxes in quantity for *anything* (although I'm sure there are a few corner cases), but especially for lightly used low-end servers and CPU-intensive computational work. Blades give you 1.5x - 2x the density if you actually need processing power, and virtualisation gives you around an order of magnitude better density, with 15 - 20 low-end "servers" easily doable on a single 8-core 1U machine, perhaps plus a 1U disk array, if all you need is a "bunch of boxes".

Dirt-cheap multicore CPUs and free virtualisation products have damn near obseleted the low-end 1U server market.

Re:done it

[r00t]

We got nearly 2x that number. The limiting factor was incoming power, with the need to avoid tripping circuit breakers or upgrading the utility connection. Cooling was a very serious problem as well. Some of the possible electrical upgrades (which we avoided) made the computers look really cheap. We chose tiny low-power motherboards and minimal power supplies to solve the problem.

The switches and KVMs are pricy, yeah, along with managed power and racks. It's a minor issue compared to REALLY AWKWARD (you have no idea) building upgrades.

Dell blades are a horrible rip-off. Note that Dell only advertizes the per-blade cost. They avoid telling you the cost of the chassis. I don't recall, but US$35k sounds like what we got quoted for an empty chassis. (we pretty much had to pretend we'd already bought the blades to get that quote!) If you're thinking about Dell blades, I think you need to investigate a bit more.

We're not compute-bound or network-bound usually, though it can happen. Often we're disk-bound with low-value temporary files.

Re:done it

[drsmithy]

We got nearly 2x that number. The limiting factor was incoming power, with the need to avoid tripping circuit breakers or upgrading the utility connection. Cooling was a very serious problem as well. Some of the possible electrical upgrades (which we avoided) made the computers look really cheap. We chose tiny low-power motherboards and minimal power supplies to solve the problem.

The only equation swings further and further in favour of blades and virtualisation the more machines you're talking about. Low-end 1U servers really only make sense if you're buying a small number of them, or if virtualisation raises other problems (eg: Oracle Appserver is not supported for production use in virtualised environments).

Dell blades are a horrible rip-off. Note that Dell only advertizes the per-blade cost. They avoid telling you the cost of the chassis. I don't recall, but US$35k sounds like what we got quoted for an empty chassis. (we pretty much had to pretend we'd already bought the blades to get that quote!) If you're thinking about Dell blades, I think you need to investigate a bit more.

I've just spent several weeks researching blade servers and dealing with competitive bids. Admittedly this was in Australia, but I sincerely doubt the numbers are (relatively speaking) any different in the US (and I say that with some background knowledge - we have a couple of IBM Bladecentres in a US colocation).

Dell's blade servers are - *by far* - the cheapest (of Dell, IBM and HP) on the market. The chassis is only about AU$8k, and that was including two switch modules. IBM's and HP's were both around twice as much, although this is somewhat offset by their higher density (Dell 7U/10 servers, IBM 7U/14 blades, HP 10U/16 blades). Density wasn't one of our primary requirements, however (we have an on-site server room and lots of rack space, power and cooling), so that wasn't really relevant.

Then there's the blade servers themselves. Again, Dell's are significantly cheaper (over 50% less by list price, around 20% less after discounting). We paid about AU$9k each for our 8x2.33Ghz, 16G RAM blades. Dell blades also had a couple of notable superiorities over IBM's, by virtue of 8 DIMM slots, hot-swappable drives and SATA drive options. HP's blades are nicer than both technology- and design-wise, but we couldn't justify the additional cost for the extra features that were mostly fluff in our environment (would have been different if we were buying for a remote datacentre - HP would be a clear winner).

I'm not throwing around idle comments here. I've just spent significant time and effort doing the numbers for our company. As I said, it's difficult to find any reason why anyone would be buying 1U servers in bulk (of any sort, but especially low-end ones), without some sort of unusual circumstances. Blades are significantly more cost-effective, energy and space efficient and have substantial manageability advantages. Going back to the hypothetical 40 VMs on 3 blade servers scenario, including storage, you'd be looking at about 1.5kW in power draw, which effectively works out to around 37W per "server". I sincerely doubt any 1U boxes are going to come close to matching that. Heck, a Mac Mini would probably have trouble matching it.

It's difficult to imagine even Google are still pursuing their 1U-server model. Multicore - especially quad-core - CPUs have changed the landscape significantly in the last 12 months or so.

Re:done it

[r00t]

The only equation swings further and further in favour of blades and virtualisation the more machines you're talking about. Low-end 1U servers really only make sense if you're buying a small number of them, or if virtualisation raises other problems (eg: Oracle Appserver is not supported for production use in virtualised environments). When you buy in bulk, you can get lower prices. That helps the 1U solution a lot. We don't pay list price.

We sometimes virtualize, usually 1-to-1 when we need to quickly change the OS we're using, but rarely a tad more. Rarely we run software on both host and guest, such as when doing client-server software testing. The big RAM sticks are too expensive to make serious virtualization practical. Virtual machines mean active swapping; our workload often keeps the disks busy even without swapping.

We've done 3 big buys, approximately: 40 machines over a year ago, 60 machines about a year ago, and 80 machines that should be arriving soon. All of these were 1U. This last time (and maybe the time before) we investigated the blades, and could only conclude that they were a rip-off.

I suppose the blades come with built-in staged power-on? We got staged power-on to avoid tripping circuit breakers.

I guess one of the things going against virtualization is that our workload is perfectly even across the machines. When we start a job, we farm it out to everything. We don't have near-idle machines ever. We might be CPU-bound, disk-bound (perhaps via swap) or even network-bound, but it's the same for every machine. I take it that this is not the case for you? You're renting out servers? Maybe you do web sites?

(BTW, if you have many thousands of near-idle servers then you ought to at least investigate IBM's big iron stuff - the virtualization there is really low overhead)

Why I don't deploy Xen

[sjames]

It's very simple really. The best virtualization solution for any given problem is the lightest weight solution that can still get the job done. The actual need (other than cool factor) to virtualize the hardware for multiple OSs is not a common one. Most of the time, UML or various patches to create jails under a single Linux kernel are good enough and certainly lighter weight and for some purposes a lot more flexible.

VMware is successful mostly because there is no lighter weight solution for Windows. There is no user mode Windows and I doubt there ever will be. Certainly nobody will be releasing a patchset to partition administrator privileges, IP addresses, etc. on Windows. OTOH, Xen has to compete w/ vserver and UML amongst others. It only gets used when a lighter weight solution just won't do.

That's certainly not a knock on Xen. When it IS necessary, it's good that it's there.