Xen Security Issue Patched

An anonymous reader sends in word of a privilege escalation security issue identified in the open source Xen hypervisor. Xen has issued a hotfix and urged all users to install it. The problem was disclosed by Secunia last week. A user of a guest domain with root privileges could execute arbitrary commands in domain 0 via specially crafted entries in grub.conf when the guest system is booted.

Comments

Already fixed in some distributions

[Anonymous Coward]

Why are vulnerabilities Slashdot-worthy? They happen almost daily, and are usually fixed quickly. This has was fixed in RHEL yesterday:

https://rhn.redhat.com/errata/RHSA-2007-0323.html

CentOS already carries this fix too.

Re:Already fixed in some distributions

[ThirdPrize]

It all depends who is editor at the moment.

Re:Already fixed in some distributions

[kscguru]

This one is newsworthy because:

1. If the guest can take over the host, this is an EXTREME vulnerability. VMs are used for security research sandboxes, intrusion containment ... hosting providers tend to sell root access to Xen guests. The Xen privilaged parts (e.g. dom0) tend to run within trusted networks. This is as bad as a remote root hole - probably worse, because it can affect a lot more machines. See a write-up about a Gartner report fearing exactly this sort of hole.
2. Compare "lots of holes, usually fixed quickly" with "very few holes". Vulnerabilities should not exist in this sort of software, no matter how quickly fixed. Would you use SSH if vulnerabilities "happen almost daily, and are usually fixed quickly"? The rate of occurrence of bugs like this is indicative of code quality. Don't be an apologist where security software is concerned.

Re:Already fixed in some distributions

[raju1kabir]

Does the default Debian Etch installation activate pygrub? I don't see any mention of it in any of the files in the xen config directory.

Re:Already fixed in some distributions

[cerberusss]

No, it doesn't. Also, it's coded by someone from RedHat:

#
# GrubConf.py - Simple grub.conf parsing
#
# Copyright 2005-2006 Red Hat, Inc.
# Jeremy Katz <katzj@redhat.com>

Re:Already fixed in some distributions

[stevey]

No. When the Debian Xen packages are installed no guests are created.

For pygrub to be used you must have it explicitly enabled in the configuration files for the Xen guest, located beneath /etc/xen.

It will only be used if you've manually created a new Xen guest and chosen to use pygrub. Many (most?) Debian users create their new Xen guests via my xen-tools software and that doesn't even support pygrub setups at the moment...

If you see no mention of pygrub in /etc/xen/* then you're almost certainly safe.

Re:Already fixed in some distributions

[raju1kabir]

Thanks very much to both of you who responded.

Re:Already fixed in some distributions

[Anonymous Coward]

If the guest can take over the host, this is an EXTREME vulnerability. VMs are used for security research sandboxes, intrusion containment ... hosting providers tend to sell root access to Xen guests.

And hosting providers tend to dissalow their users to install their own kernels, and don't use pygrub. Anyway, I agree that this shouldn't happen.

Local root vulnerabilities happen often too, and some major hosting providers give their users machine access. Should all these vulnerabilities be reported on Slashdot too? Can't administrators read security lists?

Re:Already fixed in some distributions

[Anonymous Coward]

Thanks for answering this, I can rest a little bit easier tonight.

And thanks for writing xen-tools also! It's been a big help setting up my Xen domains :)

Oughtta teach 'em

[Anonymous Coward]

What a braindead idea to put the bootloader OUTSIDE the VM. Whoever came up with that must have had no clue about the purpose of VMs, whatsoever. Grrr.
Time for someone to port Grub to run INSIDE a Xen VM.

Re:Oughtta teach 'em

[Anonymous Coward]

I believe Xen's VM model isn't to try to completely emulate a CPU as other VM products do (QEMU, VMWare, etc.), but instead only abstract it somewhat. I don't think it gives you complete ability to mess with physical memory and arrange it however you please. As such, it gives you a "hypervisor API" for requesting physical memory and controlling its virtual memory mappings.

In an environment where you can't directly mess with physical memory, you can't really have a bootloader, since a bootloader pulls data from the disk (the kernel, initrd, etc.), using either some BIOS calls or direct hardware access (which will be limited to whatever the bootloader knows how to deal with), places that data in memory, and jumps into it. This sounds like a possible disadvantage, but it has the possible benefit of not requiring anyone to mess with setting up real-/protected-mode, dealing with funky BIOS issues (which may differ among different PC models), and having to use BIOS calls to set up the environment before the real kernel boot.

In the Xen-VM world, hardware access is abstracted, and the kernel and other necessary code is already loaded for the guest VM, by the host OS.

Someone correct me if I'm wrong about this (especially the first part about physical memory access)... I can't find the documentation at the moment.

Re:Oughtta teach 'em

[InsaneGeek]

Ummm... you are incorrect regarding VMware CPU emulation and being able to have a bootloader without having direct access to physical memory.

VMware does not do any CPU emulatation at all. The physical CPU underlying it is visible to any guest, including all of the CPU's capabilities (i.e. MMX, SSE, etc). VMware is acts as scheduler for the CPU, each of the guest VM's are merely programs running inside VMware. VMware additionally does emulate particular pieces of the hardware: bios, motherboard, NIC, cdrom/floppy, USB. Guest memory is not virtualized, but is like any app that has mapped a chunk of RAM (where possible ESX does something similar to shared memory where chunks of memory are the same between guests).

A VMware guest does not directly access memory (I'm assuming you mean that it doesn't need to go through any memory manager) and it uses whatever bootloader the guest has on it's disk (grub, lilo, Windows, etc). So your guestimation is incorrect. VMware does emulate a standard bios available on motherboards (phoenix I believe) allowing the bootloaders to work without any hoops at all (I only have to go into that BIOS if I want to change from boot from disk to boot from USB/CDROM/etc, new versions make that as a prompt so I don't have to get in at all anymore). The guest doesn't need direct access because the memory starts for the guest logically at hex 0000, rather than whatever offset it actually is (0FE3). So the bootloader in the guest pulls in the data into it's logical block 0000 and jumps to it at 0000 (it doesn't know about all the rest of the memory before or after it). This is also true for the paravirtualized VMware 6.0 workstation using the new kernel paravirtualization module paravirtopts (that XEN is moving to as well) basically doing something very similar to XEN in that the guest kernel has knowledge it's in a virtual environment and can access things more directly. This is very similar to the above hardware statement, the VMWare memory scheduler makes available to a guest process a chunk of memory, just like with any process that uses a memory scheduler you don't really care about where the memory is only that you have bytes of memory starting a byte 0 going to byte , your program doesn't know that in reality it starts at address 0FE3 in the giant pool of memory the system has. This is the great secret about virtualization just like on unix everything is accessed like a file (memory, NIC card, keyboards, etc are all accessed like a file: open, read, close), everything virtualized is just a program running that is getting scheduled, inside your program it does things.

In a Xen-VM world the guest still runs a separate kernel, if it is paravirtualized the guest & the host need to use the same kernel XEN API (which in the past has basically meant that they had to be the kernel rev as the API was modified with almost every release). Something like Virtuozzo or Solaris domains would be like what I think you are meaning where there is one kernel running and it loads all code for the guests ahead of time and launches the guests into "jails" that interface back to the main kernel (they can have their own libraries, etc but everything uses the one kernel)

Re:Oughtta teach 'em

[Russell Coker]

The traditional operation of VMWare has been to simulate (in software) operations that are not permitted by hardware (for example requests to change virtual memory mappings). The default mode of operation of Xen is to have the Xen hypervisor expose an API for Xen enabled kernels to use to request such operations (it's basically a system call to change memory mappings etc).

I believe that recent versions of VMWare and Xen support the virtualisation features of the latest Intel and AMD CPUs so they can run unmodified OSs (without special Xen kernel support) at full speed (software emulation of protected-mode instructions is very slow - the fact that such instructions are not used often means that overall performance may not suffer much). But not owning a machine with such hardware I have not had a chance to test this.

The Dom0 in Xen provides the kernel and initrd to the Xen system for booting, the process which does this runs as root and if it is compromised then you lose.

re:Xen Security Issue Patched

[Anonymous Coward]

Maybe now we won't get any more of those theoretical physicists lurking around now.

Xen rooted

[Mister Liberty]

Xen and the art of powercycle maintenance.

Goodbye Gordan!

[Anonymous Coward]

Thank goodness this was fixed. No longer will we have to worry about bearded miscreants sneaking into Xen and shooting the place up!

Re:Too much Xen, too little boobs

[Anonymous Coward]

Boobs are a scary security issue all right. Especially alien boobs.

Now, go back to your xbox kiddo.

OT: I lolled at the CAPTCHA, 'twas "cherish"...

Have there been many issues with ESX?

[Anonymous Coward]

This is a moderately scary security issue -- a few of these per year and the current virtualizing trend might start to unravel.

The XenSource code base (well, some of it at least) is Open. Its competitor, ESX from VMware, isn't. Philosophy aside, how has ESX been doing on the security front?

Re:Have there been many issues with ESX?

[cerberusss]

This is a moderately scary security issue -- a few of these per year and the current virtualizing trend might start to unravel.

You're quite right; funny thing is, this is described as "less critical" by Secunia because they say it's not a remote exploit. You have to be in a virtualized machine to start with and they don't call this 'remote'.

Re:Have there been many issues with ESX?

[essdodson]

How would we know? VMWare hides what their updates are really about.

Re:Have there been many issues with ESX?

[weicco]

What if virtual machine uses Plan9 and attack is initiated from remote machine across the network, would that count as remote hole? :) I'm not sure if that's even possible though...

but but yesterday...

[McNihil]

we were told that it was not possible to do a VM root kit...

Now that that pdf from both Xen and VMWare read like someone trying hard to sell a product is another thing.

Re:but but yesterday...

[Anonymous Coward]

Assuming you're not trolling, as sadly, many slashdotters lack basic reading comprehension.

We were told the day before yesterday that VM-Based Rootkits Proved Easily Detectable, not impossible.

Use emulation?

[TheLink]

I believe the bootloader only runs for a short while so maybe they can use emulation for running the bootloader stuff safely.

Then switch to "native" once you hit the kernel stuff etc.

Probably a lot more work tho.

Wow, that's amazing!

[Kingrames]

A security issue fixed in the state of Zen?

That's amazing. Chalk another victory up for the pocket-protector crowd. Is there anything they can't do?

xensource.com uses IIS - How lame!

[T.Bickle]

Looks like they are not too concerned with security either way:
Server Microsoft-IIS/6.0
X-Powered-By ASP.NET

Re:xensource.com uses IIS - How lame!

[Anonymous Coward]

"lame" only to the retardeds that don't realize IIS 6.0 != IIS 5.0

Re:xensource.com uses IIS - How lame!

[ArsonSmith]

What? IIS is no longer "the laughing stock of security."

Re:Additional details

[raju1kabir]

If there's any useful information on that site it's well-hidden. A search for "pygrub" turns up zero results. I suspect you are just spamming.

BLACK MESA CORPORATE EMAIL

[Tyrsenus]

Xen Security Issue Patched: The Black Mesa science team successfully closed the portal that was inadvertently opened this morning during the resonance cascade scenario.

Customer of mine...

[cerberusss]

Cool to see this on Slashdot. The guy who found the vulnerability is actually a customer of mine. I recently started a business in hosted Virtual Private Servers. Joris van Rantwijk, the bug reporter, was interested to become a customer and I said why don't you try it out for a few weeks?

As a plus point, I let them boot their own kernels (I trust my custommers). Next thing I know, he tells me to check my /root directory ON MY PHYSICAL MACHINE (i.e. domain 0 in Xen speak) where I find a file describing the exploit...

Oh don't bother to check out my business' website, it's not translated yet in English... (I'm Dutch).

why would a domU have grub anyway?

[myowntrueself]

A paravirtualised domU does not need a boot loader and does not even have access to its own kernel. Which is nice because you can give it an unmodular kernel which is more secure in as much as a user on that domU cannot load their own (potentially exploiting) kernel modules.

Its only an HVM domU that would need a boot loader and its own kernel.

I've read through TFA and the Xen mailing list and I can't see anything that says whether this affects both paravirtualised *and* HVM or just one or the other...?

In the case of paravirtualised, why on earth would the creating the domU even *look* for a /boot/grub/grub.conf in the domU filesystem?? Makes no sense at all.

Re:why would a domU have grub anyway?

[demon]

Paravirtualized domUs can be booted by using a pseudo-bootloader called pyGrub, which mostly emulates GRUB in its functionality, allowing a self-contained domU image to self-bootstrap (no dom0-hosted kernel image is needed). Unfortunately the script runs within dom0, which is where the issue comes in - since it's executing within that domain, and with root's privileges (it has to be able to examine the root block device), I guess it was only a matter of time till someone figured out how to make it do... things.

Re:why would a domU have grub anyway?

[myowntrueself]

Unfortunately the script runs within dom0, which is where the issue comes in

And so dom0 gets to run a script against content in a file in domU...

My God thats just insane and is *asking* for trouble!!!

What *moron* thought this would be a Good Idea???

Because RedHat is a bunch of idiots

[Anonymous Coward]

You're describing what I consider to be the right way to do things.

However some vendors shipping Xen *cough*redhat*cough* have this brain-dead idea that the VM should look exactly like a real machine, including bootloaders and all accompanying nastiness.

To that end, they created "pygrub", which expects the VM's disk image to have a boot sector, whereby it copies the kernel image and initrd to a temporary file, and then passes that to xen to start the domain (my guess is that this is where the exploit lies.)

...words..stuff...

[Joe U]

A user of a guest domain with root privileges could execute arbitrary commands in domain 0 via specially crafted entries in grub.conf when the guest system is booted.

Seriously, guys, Star Trek:TNG is off the air. You can stop writing like this now.

The summary in Haiku

[CCFreak2K]

There was an exploit
and then it was fixed
slow news day

Avoid PyGrub and everything is fine

[Russell Coker]

PyGrub expects that the root device is partitioned and searches on partition 1 for the GRUB configuration. This is bad because it requires you to have a partition table which makes things inconvenient if you want to resize the disk space provided to the virtual machine.

I run my Xen DomU's with a single virtual disk per filesystem, no partition tables (the DomU mounts /dev/vbda or /dev/hda etc) and I can easily resize any filesystem at any time by stopping the DomU, extending the LVM volume, and extending the filesystem.

This doesn't work with PyGrub, I could fix the bug but I haven't bothered - it's easier to just have the kernel and initrd stored outside the filesystem managed by the DomU.

This configuration makes it easier to manage (one kernel and initrd for multiple DomU's) and avoids security issues with PyGrub.