Slashdot Mirror
Desperately Seeking Xen
AlexGr sends us to an excellent article on the state of Xen by Jeff Gould (Peerstone Research). He concludes that the virtualization technology has some maturing to do and will face increasing competition for the privilege of taking on VMWare. Quoting: "What's going on with Xen, the open source hypervisor that was supposed to give VMware a run for its money? I can't remember how many IT trade press articles, blog posts and vendor white papers I've read about Xen in the last few years... The vast majority of those articles — including a few I've written myself — take it as an article of faith that Xen's paravirtualizing technical approach and open source business model are inherently superior to the closed source alternatives from VMware or Microsoft."
It is true that Xen requires special hardware to legally run MS Windows. It is also better for performance, generally, to have such hardware. However, there is nothing stopping you from running Xen on pretty much any computer you are likely to own as long as the VMs are Linux based.
Not true.
If you have VT-capable hardware then you can run Windows under Xen. You do need the hardware to support it though, and that is a problem for some home users. Recent AMD and Intel chips have slightly differing VT support but both work.
I run Xen at home along with xen-tools (which I wrote) to easily create new Debian guests on demand. These are used for software testing, hacking, and general service isolation.
I think Xen is just now reaching "mainstream" in the sense that you don't have to be an early adoptor or major tinkerer to get it working. Now that distributions are including Xen kernels in their newer releases it really us available for all.
Not only that, but I've been running it in a production environment for about a year and I'm about to deploy a HUGE set of servers as VMs using it. Xen beats VMware in one arena: price. If you use the open source version (which I'm doing) it's free. Only VMWare's ESX can compare to Xen. And unlike some people here have been saying, you DON'T need a special processor for Xen unless you plan to virtualize Windows. In my environment, I'm only virtualizing Linux, so I can use regular x86 CPUs dating back to 1998 for Xen. The only exception is the deployment of Zimbra I'm going to do. It requires Redhat Enterprise Linux 4 and NPTL, so I can't run it paravirtualized, it must run HVM which requires the special processors. However, who today isn't getting new hardware with HVM support?
Currently my two Xen servers here at work serve out about four VMs (all paravirtualized on older hardware) for critical and I/O intensive tasks like proxy servers for nearly 1000 machines, or the firewall syslog server for a dual T3 link with about 5000 users behind it sucking the bandwidth dry. So you can't claim it doesn't perform either. Now, if you want point and click administration and an easy set up, then yeah, Xen is behind the times. But performance wise it's leaps and bounds above VMWare. Trust me, I was a VMWare fan before you were in virtualization diapers. And I still am for some applications. But for places where I need something to be cost effective AND give me the features of VMWare ESX, Xen is the ONLY answer.
-"...bad old ideas look confusingly fresh when they are packaged as technology" - Jaron Lanier (Digital Maoism on Edge.o
Another question hanging over Xen performance concerns the availability of paravirtualized drivers for Windows.
This isn't true completely. The problem is you cannot get these drivers by downloading the OpenSource Xen. You MUST buy the XenSource version. If you run Windows on the *complete* open source version, your network throughput is going to suck like you would not believe. You have to use the XenSource version to get the paravirtualized drivers that bring the network performance closer to what it should be. Virtual Iron has a set of drivers also. (which I believe are better than Xen's, but don't hold me to that)
I found a lot of great insight about virtualizing from Xen to VMWare to Virtual Iron and others on this site. http://ian.blenke.com/xen
Yes, but as he already told you, what you know is out of date. You can run an un-modified Windows on top of Xen provided your hardware supports the Intel or AMD VT extensions.
If you want to get a colorful thread of comments started on slashdot, there are 3 ways to do it with guaranteed results:
1) Say something bad about linux (or about Apple).
2) Say something good about Microsoft (or about Apple).
3) Throw a grenade in the room about Open Source software like this:
The vast majority of those articles -- including a few I've written myself -- take it as an article of faith that Xen's paravirtualizing technical approach and open source business model are inherently superior to the closed source alternatives from VMware or Microsoft.
I'm not making any value judgements here--I'm just amused.
I might know what I'm talkin' about, but then again, this is Slashdot...
The only exception is the deployment of Zimbra I'm going to do. It requires Redhat Enterprise Linux 4 and NPTL
Last I checked, Zimbra runs on Ubuntu 6 just fine.
It depends on what I'm doing. If you weren't trying to be cute, I'd say you were trolling. In reality, it's very common practice to use LVM to clone a filesystem, make some changes to the various files that set IP and hostname as well as other unique host settings and bring up alternate "Test" VMs on a Xen box. So some days I might be running three VMs other days eight or ten. It all depends on what I need to do.
As an aside, I forgot to mention that there are NO other products other than VMWare ESX that offer "live migration" of a running VM from one hardware host to another. That's right... you can take a VM that is running with many users actively using it and move it from one physical box to another with only a few milliseconds down time. The users NEVER notice. The free VMWare server can't do that. Micrsoft's Virtual Server can't do that until they have a hypervisor. And there really isn't anything else that can.
-"...bad old ideas look confusingly fresh when they are packaged as technology" - Jaron Lanier (Digital Maoism on Edge.o
I have been trying to use Xen at home to test it out and compare it to VMWare, which I've used at work. Once you manage to get Xen clients working, it's fine. It does a good job of running VMs, and can be used to partition resources on a powerful machine.
But, the main problem is the steep learning curve for getting Xen running in the first place. The (python based) management GUIs included with Fedora or Ubuntu are weak at best (although, slowly improving.. the UI in Fedora 7 does manage to make setup easier than the command-line alternative). The ongoing management / monitoring of VMs is okay, but weak in comparison to VMWare.
There are also a lot of little quirks in Xen. Installing Win2k in a client VM required a lot of searching for how to attach an ISO image to a running VM (it's not a simple GUI operation like in VMWare/Parallels/VirtualPC, it requires a terminal command with unintuitive options, which never worked for me.. I finally dug out my CD and got the physical CD drive to attach to the VM). Windows VMs have an odd issue where the mouse pointer is offset form the actual pointer (it's a known issue, and is helped by turning off mouse acceleration in Windows preferences, but it is still a problem). Installing client VMs can be challenging.. Ubuntu feisty wouldn't install until I set the VM as a Solaris client, and after a few other tweaks it finally installed and worked fairly well.
Most of the Xen problems are solvable, after playing with command-line tools, figuring out poorly documented parameters, and lots of googling. At the end of the day, it's one of those "Xen is free, if your time has no value" type things. VMWare Server is probably a better option if you just want it to work for home/free uses. For commercial use, VMWare ESX Server is the way to go. It has simple VM setup for many client OS's, excellent management of large groups os Hypervisors and virtual machines.
The commercial alternative from XenSource (free to use, but limited to 4 VMs; or less restricted versions for increasing $$) offer a better management UI, but are too restricted for my taste. The management app is much better, but not as good as VMWare.. If I'm going to pay for one, I'll go for the best option.
The problem with giving access to hardware to guests at the moment is that without an IOMMU, any DMA request the driver issues will read or write memory from a physical address indicated by the driver. In a virtual machine, what the driver thinks is a physical address is actually a virtual address. This means a DMA request will read from or write to an arbitrary memory location. By putting the HVM guest at the start of memory, this translation is the identity function, so the driver will work. The only downside is that you lose protection from other domains; a malicious driver can still damage your other VMs or even the hypervisor.
I am TheRaven on Soylent News
The problem with Hardware acceleration in VMS is fairly straightforward. The driver sends information such as 'use bitmap located at pos x in memory' The way memory mapping works, the VM might be given a chunk of memory (i.e. positions 100 to 200) and sees this as 000 to 100. for the VM, x = 010. When the card tries to access that memory, it's memory that might be assigned to a different VM, and thus garbage. Unfortunately, this generally requires the cooperation of the drivers.
fortunately, VirtualGL is working on a generic fix, although it's still immature.
"Hate is baggage. Life's too short to be pissed off all the time." Danny Vinyard -American History X
Expensive? Not really, if you compare the costs of actually getting that number of servers. Given the feature set you get, it's pretty modest. If you work in an educational setting, it's even cheaper. You can get VI3 Enterprise and a tier one server - 2U rackmount, dual quad-core system (2.4 ghz) with 16GB of RAM, dual power supply, 6 hour CTR service, an additional NIC and a 4GB fibre channel card and about 512GB of local storage for about $16,000. Depending on the size of the VMs you need to run, you can easily get anywhere from 12 to 20 VMs on that server.
We're entering our fourth year of server virtualization under VMware, and we would've sunk without it - trying to meet customer demands in our space and budget would have been impossible.
Factoring in things like the cost of gigabit ports on a data center switch, cost of power outlets (distribution is often as much of a problem as capacity), and the cost of fibre channel switch ports, there's a huge savings per server. We've got 42 VMs running on three servers similar to the above (one 2xdual core, two 2xquad) and an older server still running VMware ESX 2.5.x slated for replacement next year.
And yes, fibre channel. It's still cheaper than 10GB ethernet, and can run at 4GB.
Xen saved my former employer a bunch of money and gained then great flexibility and reliability. They use an AoE (ATA over ethernet) SAN so the compute nodes are totally diskless and all of the data and root filesystems are on the SAN. Now they have email, database, web serving, nearly all of their critical functions in a highly available xen-aoe cluster. I am working with them to release all of the codes and configs in production and we are setting up a website at xenaoe.org (not up yet, but soon) to host the project.
Here is something I wrote up about this architecture for the company when the project went live:
What is Xen?
Xen is a free virtualization system similar to VMware but different. It allows us to run multiple servers/operating systems all on one physical piece of hardware while providing isolation between them.
What is AoE?
AoE is a SAN technology. Similar to Fibrechannel (but far less expensive) or iSCSI (but far simpler and more efficient).
What are the advantages of Xen and AoE for our company?
Xen allows us to more efficiently utilize our hardware resources. The majority of cpu power on your average computer goes unused. Even on servers. They just sit there waiting for something to happen. Even if we get a web request every second the time between one request and the next is an eternity for a cpu running at 2 gigahertz. But powerful cpu's are needed for those short bursts of activity. By using Xen to run multiple servers in their own domains (areas of memory) completely isolated from each other on the same physical hardware we can squeeze more utilization out of our existing CPU's/servers. This means we can get by with fewer CPU's, less rackspace, use less power, and require less air conditioning. By encapsulating the servers into this sort of infrastructure it also allows enhanced management capabilities by allowing the administrator (such as myself) to be able to get console access on the server or restart the server while remote instead of having to drive to the datacenter (which in our case is a 30 minute drive down to Kearny Mesa).
AoE allows us to put a bunch of disk in relatively inexpensive and low CPU powered servers on the network and allow the rest of the servers to access it exactly as if the disk were locally installed in that server. This is advantageous because we can now aggregate all of our disk into one system and treat it like a pool of storage where we can dole out an appropriate amount of disk to each server (often only 10 or 20G is needed) instead of having to put in a dedicated 250G disk which is the minimum you can easily buy these days and waste a lot of disk and power to run it.
The combination of Xen and AoE allows us all of the above plus some interesting fault tolerance abilities. There are now two levels of redundancy in our disk systems and an extra level of redundancy in the cpu's also in that if one cpu fails (or the associated motherboard, RAM, or network card) we can easily switch the servers that were hosted on that machine over to another cpu on the network with either zero or very minimal downtime whereas previously that kind of failure would have required me to drive down to the datacenter and shuffle hardware around or buy new hardware to replace the failed system which all takes time and can result in prolonged downtime.
Yes, you're wrong. You can set up iSCSI over Gb ethernet using nothing more than a normal Linux box with normal disks.
You can even have a pair of servers with a GigE crossover running NBD to provide 'network RAID 1' of the disks with transparent hot failover.
Indeed, why would they?
Because being able to dedicate a "machine" to each service rather than trying to run dozens of different services on the same machine vastly simplifies operations.
Pay attention now. This is not how Google runs their datacenter.
How Google runs their datacentre is not relevant to most people, who have vastly different requirements, budgets and capabilities.
Shop around. One can find power-efficient 1U boxes. Sometimes non-rackmount is better, including weird stuff like the Mac Mini. Be willing to look beyond Intel and AMD. VIA makes some low-power chips.
Indeed. Then instead of the 2-4 power connections, ethernet connections, fibre channel connections, the cooling capabilities, electrical capacity, 4RU (or less, with other hardware options) of rack space, etc you need for a single machine running 30 VMs, you need *60* power/ethernet/FC ports, higher cooling and power needs, 30 rack units, 30 KVM/serial/RJ45 ports, etc.
Price those out in a datacentre and suddenly that "expensive, single point of failure" becomes cheap and easy to turn into 3 or 4 machines running VMs.
That's just the basic physical footprints - this is before even getting into the _manageability_ advantages of VMs over physical machines (eg: being able to roll out new servers by running an install script and coming back 20 minutes later, instead of having to get physical hardware specced, ordered and installed).