The Xen PV drivers have historically been closed source for Windows. Fortunately a brave soul in the community stepped up and wrote a set of GPL drivers but Citrix still maintains their proprietary drivers. In general, there's a great deal of fragmentation with Xen PV drivers because they haven't been Open Source from the start.
I think the fact that KVM is avoiding this is quite good.
I've always wondered how paravirtualizing some functions such as I/O or networking affects security.
Say a VM gets compromised, and is able to do what it wants with the block devices, how tough would it be to get out of the VM? If malicious code is able to access the host's block device that runs in kernel mode and start running code directly on the host's OS, game over.
Unlike Hyper-V and Xen, in KVM a paravirtual device looks an awful lot like an emulated device. For instance, virtio-net appears to the guest as a normal PCI device. It's quite conceivable that a hardware vendor could implement a physical virtio-net card if they were so inclined. In our backend, we implement virtio-net like any other emulated device.
This means from a security perspective, it's just as secure as an emulated driver. It's implemented in userspace and can be sandboxed as an unprivileged user or through SELinux.
VMware uses a similar model. Hyper-V and Xen prefer to not model hardware at all and use special hypervisor-specific paths. From a security perspective, the fact that these devices are on a different code path means that they have different security characteristics than emulated devices. For instance, in Xen, a paravirtual network device is backed directly in the domain-0 kernel so an exploit in the xenpv network device is much more severe than an exploit in a Xen emulated network device (since the device emulation happens in an unprivileged stub domain).
CONFIG_HW_RANDOM_VIRTIO enables it. It's been there for quite a while.
We could easily support it in KVM but I've held back on it because to really solve the problem, you would need to use/dev/random as an entropy source. I've always been a bit concerned that one VM could starve another by aggressively consuming entropy.
No, SMM is loaded from ROM memory by the BIOS. You would have to reload the SMM code every time.
What's more, this only works while the SMM code stays resident in the CPU cache. You would need something running at the OS level that was constantly rewriting this memory to ensure it stayed in the CPU cache.
I expect this would actually be quite difficult to build a root kit with that was not as easy to detect as any other root kit.
Both Nehalem and Barcelona (Phenom) are quad-core and most importantly, support EPT and NPT respectively. This feature has significant impact on virtualization performance.
If you want to run 4 VMs, you'll probably want to have a fair bit of memory. 4GB would be good, 8GB would be better.
Is there any reason you couldn't keep a list of processor dependent memory locations and regenerate them for the current machine as part of the migration?
The new Intel/AMD CPU features that allow masking of CPUID bits while running virtualized also make processors recent enough that most of the interesting features are present - MMX, SSE up to ~3. The "common subset" ends up looking like an early Core2 or a Barcelona (minus the VT/SVM feature bits, of course) - Intel and AMD run about a generation behind on adding each other's instructions. Run on anything older than the latest processors, and you have to trap-and-emulate every CPUID instruction. Enough code still uses CPUID as a serializing instruction that this has noticeable overhead.
Modern OSes do not use CPUID for serialization. We trap CPUID unconditionally in KVM and have not observed a performance problem because of it. Older OSes did this but I'm not aware of a modern one.
My understanding of the reason for the recent CPUID "masking" support is because if you are not using VT/SVM (Xen PV or VMware JIT), there is no way to trap CPUID when it's executed from userspace. AMD just happened to have this feature so when Intel announced "FlexMigration", they were able to just document it. I don't think it's really all that useful though.
(As a side note to everyone reading, the reason Linux timekeeping is such a problem is that TSC issue. Intel long ago stated TSC was NOT supposed to be used as a timesource. Linux kernel folks ignored the warning, made non-virtualizable assumptions, and today are in a world of hurt for timekeeping in a VM. And only now, many years later, are patching the kernel to detect hypervisors to work around the problem.)
The TSC is often used as a secondary time source, even outside of Linux, but yes, Linux is the major problem. But Windows it not without it's own faults wrt time keeping. Dealing with missed timer ticks for Windows guests is a never ending source of joy. Virtualization isn't the only source of problems here. Certain hardware platforms have had overzealous SMM routines and the results was really bad time drift when running Windows.
Declaration: VMware support engineering here, but speaking strictly on my own behalf.
The stability issues are justified if you consider all types of VMs. Windows 2003, default RHEL5 kernels etc use more than the basic set of assembler instructions (disk IO code uses MMX, SSE etc).
KVM goes to great lengths to by default, mask out CPUID features that aren't supported across common platforms. You have to opt-in to those features since they limit a machine's migrate-ability.
However, I won't say this is always safe. In reality, you really don't want to live migrate between anything but identical platforms (including identical processor revisions).
x86 OSes often rely on the TSC for time keeping. If you migrate between different steppings of the same processor even, the TSC calibration that the OS has done is wrong and your time keeping will start to fail. You'll either get really bad drift or potentially see time go backwards (causing a deadlock).
If you're doing a one time migration, it probably won't matter but if you plan on migrating very rapidly (for load balancing or something), I would take a very conservative approach to platform compatibility.
I live in New Jersey and am annoyed to no end that we have to pay to use the beach.
It's for good reason. The individual communities are responsible for maintaining the beaches. This includes cleaning (people leave a ton of trash on the beach), life guards, and sand. Most of the communities along the Jersey shore lose a lot of sand to erosion during the winter and have to periodically buy sand to put on the beaches. This all costs money. There's no reason why only the residents of the communities should bear the cost of this.
Does that same argument apply to VMWare too then? (for their product that allows you to fully virtualize non-modified operating systems on non VT/SVM hardware)
No. VMware uses dynamic translation which is a complete mechanism. VirtualBox uses binary patching which severely violates correctness.
It seems that VirtualBox.org's product, fully virtualizing a copy of XP on my non-VT machine under a linux host OS, totally runs circles around Xen even on VT hardware as far as performance is concerned. Integration into the host enviroment is also quite beautiful. Why is there seldom a mention of VirtualBox in this arena?
Because the mechanism they use to virtualize on non-VT/SVM systems is incorrect. Since a guest will likely eventually crash, it's not interesting for anything but hobbyists. They technique they used is fundamentally broken and will never be fixable so it's just not on people's radar. The same applies to kqemu.
As I understand it, when VT/SVM is present, VirtualBox does behave correctly. The reason it isn't interesting there is that the code is horrific and will never make it into the upstream kernel.
Some issues might be windows-specific, such as windows 2000 taking so long to turn the whole display desktop-colored (initializing video memory, I guess?) taking literally a minute or more.
That's been fixed.
And yes, by VDE I mean Virtual Distributed Ethernet.
VDE is not part of the KVM project. The best way to deal with networking is to just setup bridges yourself.
2.6.20 will be the first real release of KVM. This benchmark used 2.6.20-rc3. For 2.6.20-rc4, a new shadow paging implementation was introduced (memory virtualization) that is significantly faster than what was present in -rc3. I've only got microbenchmarks handy, but context switch time, for instance, improved by about 300%.
I suspect if they reran their benchmarks with -rc4, the KVM numbers would be much more competitive with the Xen numbers (although I do suspect Xen will still be on top--slightly).
Either you didn't pay attention or your school wasn't that good. Sorry.
A "console emulator" can be a straight forward emulator in which case, you should know enough from just your basic architecture courses. Did you discuss instruction decoding and ISAs?
Modern console emulators are probably Just In Time compilers. You should have had a compilers class and the prof should have at least mentioned binary translation. Even if they didn't, you should have spent a little time on JITs in an architecture class.
A college education is not necessarily about knowing how to solve problems, but how to decompose a problem into a series of problems that you can then figure out how to solve. For a console emulator, that may mean that first you know you have to read about the architecture your emulation (what's the ISA, what are the components, etc.). Then you realize you have to parse the actual ROMs (here's where your automata/compiler background kicks in). Then either emulate each instruction (tedious) or do dynamic translation.
If you wouldn't even know where to start, you didn't get the right education. I'd recommend trying to find a masters program or pick up some text books.
"While many users reading Slashdot embrace Windows, ZDNet is running an article on why the rest of the world isn't ready. One note for Windows developers: 'Stop assuming that everyone using Windows (or who wants to use Windows) is a Windows expert.' While a lot of these topics have been brought up as both stories and comments on Slashdot, this article pretty much sums up why Fedora Core 6 could be absolutely terrible, and people would still believe there is no other option."
If you are a Windows user (like the author of TFA), then you will be pissed off by Linux because you aren't as productive immediately (because, *gasp*, it's different).
If you are a Linux user (like me), then you will be pissed of by Windows because you aren't as productive immediately (because, *gasp*, it's different).
It's the same old story that's been told a thousand times before. Come on, it's not a slow news day. Hell just froze over.
Feel free to correct me if I've mis-understood, but your position seems to be that while it may be prohibitively hard to detect the a trojan hypervisor, it's still technically possible. Joanna presented as if BluePill were undetectable, you're contesting that it's detectable, just hard.
No, not really. I'm saying that in all circumstances, it's detectable. I'm claiming that even with all the engineers in the world working on this thing, it's still detectable (although it may be hard).
Practically speaking, building a VMM is really difficult. I have serious doubts that one could make even a reasonably complete version of this sort of thing. To make something that was more than trivially easy to detect would require an even larger under taking. And at that point, you'd have a VMM that was so good, I doubt you'd bother making malware. You'd start selling it for $800 a CPU and make yourself a fortune:-)
An excellent idea, however if AV companies start doing this, what will BluePill++ start doing? Blocking NTP?
Maybe. But then, you use some other sort of protocol. It's just like the arms race between today's virii and anti-virus software. It's pretty much exactly what we've been dealing with for years.
If I understand your method correctly, it still requires the use of an external clock. If that's the case, there's still no programmatic way to do it, you're still relying on external sources, which is something your average end-user is unlikely to do.
We're talking about anti-virus software here--not the average end-user. Anti-virus software just has to use NTP (or invent it's own time-keeping protocol) to check external time. There is of course a skew when checking time over the internet so you just make sure that your sample interval is much larger than your potential skew.
For instance, if you can only reliably get time from an NTP server plus or minus.5 seconds, and we're claiming that under blue pill, rdmsr will take 10x longer, then you need run the rdmsr instruction in a tight loop for at least.05 seconds.
Of course, you'd probably run it much longer just to be safe. For instance, if you ran it for 2 seconds, on native hardware it would take 2 seconds (and based on an external clock, you'd get anywhere from 1.5 to 2.5 seconds). Under blue pill, you would measure anywhere between 19.5 and 20.5 seconds. Such a dramatic difference would be a clear sign that you're in a VMM. That assumes that this is the ideal version of Blue Pill too which current doesn't exist. For less than ideal versions, there are much simplier tricks you can do.
She admitted that timing attacks were her weakness, as did the other guy who talked about virtualization-based rootkits. The problem is that you have to have a benchmark to compare it to, and you have to assume that the hypervisor doesn't modify the time whenever it is called.
The difference is order of magnitudes. Hypervisor exits incur penalties on the order of thousands of cycles. If you don't know the specifics of the system you're on, you can always compare the cost relative to a similiar instruction. For instance, while you may not know how many cycles a rdmsr should take, you should have a rough idea of the ratio of how many movl $0, %eax's you can do verses rdmsrs within a certain time period.
The hypervisor cannot modify external time sources because it cannot detect the use of an external time source in the general case.
The key to avoid skew is numbers. Do a ton of these operations and any potential skew will be lost in the noise.
Slashdot Mirror
The short answer is: Whatever.
It's a little more nuanced than that. To the extent that a long term session is more predictable than a short term session (or vice versa), it may matter. See Timing Analysis of Keystrokes and Timing Attacks on SSH.
The Xen PV drivers have historically been closed source for Windows. Fortunately a brave soul in the community stepped up and wrote a set of GPL drivers but Citrix still maintains their proprietary drivers. In general, there's a great deal of fragmentation with Xen PV drivers because they haven't been Open Source from the start.
I think the fact that KVM is avoiding this is quite good.
I've always wondered how paravirtualizing some functions such as I/O or networking affects security.
Say a VM gets compromised, and is able to do what it wants with the block devices, how tough would it be to get out of the VM? If malicious code is able to access the host's block device that runs in kernel mode and start running code directly on the host's OS, game over.
Unlike Hyper-V and Xen, in KVM a paravirtual device looks an awful lot like an emulated device. For instance, virtio-net appears to the guest as a normal PCI device. It's quite conceivable that a hardware vendor could implement a physical virtio-net card if they were so inclined. In our backend, we implement virtio-net like any other emulated device.
This means from a security perspective, it's just as secure as an emulated driver. It's implemented in userspace and can be sandboxed as an unprivileged user or through SELinux.
VMware uses a similar model. Hyper-V and Xen prefer to not model hardware at all and use special hypervisor-specific paths. From a security perspective, the fact that these devices are on a different code path means that they have different security characteristics than emulated devices. For instance, in Xen, a paravirtual network device is backed directly in the domain-0 kernel so an exploit in the xenpv network device is much more severe than an exploit in a Xen emulated network device (since the device emulation happens in an unprivileged stub domain).
CONFIG_HW_RANDOM_VIRTIO enables it. It's been there for quite a while.
We could easily support it in KVM but I've held back on it because to really solve the problem, you would need to use /dev/random as an entropy source. I've always been a bit concerned that one VM could starve another by aggressively consuming entropy.
lguest does support this backend device though.
No, SMM is loaded from ROM memory by the BIOS. You would have to reload the SMM code every time.
What's more, this only works while the SMM code stays resident in the CPU cache. You would need something running at the OS level that was constantly rewriting this memory to ensure it stayed in the CPU cache.
I expect this would actually be quite difficult to build a root kit with that was not as easy to detect as any other root kit.
Both Nehalem and Barcelona (Phenom) are quad-core and most importantly, support EPT and NPT respectively. This feature has significant impact on virtualization performance.
If you want to run 4 VMs, you'll probably want to have a fair bit of memory. 4GB would be good, 8GB would be better.
Is there any reason you couldn't keep a list of processor dependent memory locations and regenerate them for the current machine as part of the migration?
The halting problem?
The new Intel/AMD CPU features that allow masking of CPUID bits while running virtualized also make processors recent enough that most of the interesting features are present - MMX, SSE up to ~3. The "common subset" ends up looking like an early Core2 or a Barcelona (minus the VT/SVM feature bits, of course) - Intel and AMD run about a generation behind on adding each other's instructions. Run on anything older than the latest processors, and you have to trap-and-emulate every CPUID instruction. Enough code still uses CPUID as a serializing instruction that this has noticeable overhead.
Modern OSes do not use CPUID for serialization. We trap CPUID unconditionally in KVM and have not observed a performance problem because of it. Older OSes did this but I'm not aware of a modern one.
My understanding of the reason for the recent CPUID "masking" support is because if you are not using VT/SVM (Xen PV or VMware JIT), there is no way to trap CPUID when it's executed from userspace. AMD just happened to have this feature so when Intel announced "FlexMigration", they were able to just document it. I don't think it's really all that useful though.
(As a side note to everyone reading, the reason Linux timekeeping is such a problem is that TSC issue. Intel long ago stated TSC was NOT supposed to be used as a timesource. Linux kernel folks ignored the warning, made non-virtualizable assumptions, and today are in a world of hurt for timekeeping in a VM. And only now, many years later, are patching the kernel to detect hypervisors to work around the problem.)
The TSC is often used as a secondary time source, even outside of Linux, but yes, Linux is the major problem. But Windows it not without it's own faults wrt time keeping. Dealing with missed timer ticks for Windows guests is a never ending source of joy. Virtualization isn't the only source of problems here. Certain hardware platforms have had overzealous SMM routines and the results was really bad time drift when running Windows.
FWIW, KVM live migration has been capable of this for a long time now.
KVM actually supported live migration of Windows guest long before Xen did. If you haven't given KVM a try, you should!
Declaration: VMware support engineering here, but speaking strictly on my own behalf.
The stability issues are justified if you consider all types of VMs. Windows 2003, default RHEL5 kernels etc use more than the basic set of assembler instructions (disk IO code uses MMX, SSE etc).
KVM goes to great lengths to by default, mask out CPUID features that aren't supported across common platforms. You have to opt-in to those features since they limit a machine's migrate-ability.
However, I won't say this is always safe. In reality, you really don't want to live migrate between anything but identical platforms (including identical processor revisions).
x86 OSes often rely on the TSC for time keeping. If you migrate between different steppings of the same processor even, the TSC calibration that the OS has done is wrong and your time keeping will start to fail. You'll either get really bad drift or potentially see time go backwards (causing a deadlock).
If you're doing a one time migration, it probably won't matter but if you plan on migrating very rapidly (for load balancing or something), I would take a very conservative approach to platform compatibility.
I live in New Jersey and am annoyed to no end that we have to pay to use the beach.
It's for good reason. The individual communities are responsible for maintaining the beaches. This includes cleaning (people leave a ton of trash on the beach), life guards, and sand. Most of the communities along the Jersey shore lose a lot of sand to erosion during the winter and have to periodically buy sand to put on the beaches. This all costs money. There's no reason why only the residents of the communities should bear the cost of this.
It was the first real paravirtualization approach. Check out Xen and the art of virtualization, it's a pretty good read.
Nope. IBM's pHype, Denali, arguably, L4, Exokernel, and even Nemesis were doing it long before Xen came around.
Does that same argument apply to VMWare too then? (for their product that allows you to fully virtualize non-modified operating systems on non VT/SVM hardware)
No. VMware uses dynamic translation which is a complete mechanism. VirtualBox uses binary patching which severely violates correctness.
It seems that VirtualBox.org's product, fully virtualizing a copy of XP on my non-VT machine under a linux host OS, totally runs circles around Xen even on VT hardware as far as performance is concerned. Integration into the host enviroment is also quite beautiful. Why is there seldom a mention of VirtualBox in this arena?
Because the mechanism they use to virtualize on non-VT/SVM systems is incorrect. Since a guest will likely eventually crash, it's not interesting for anything but hobbyists. They technique they used is fundamentally broken and will never be fixable so it's just not on people's radar. The same applies to kqemu.
As I understand it, when VT/SVM is present, VirtualBox does behave correctly. The reason it isn't interesting there is that the code is horrific and will never make it into the upstream kernel.
Some issues might be windows-specific, such as windows 2000 taking so long to turn the whole display desktop-colored (initializing video memory, I guess?) taking literally a minute or more.
That's been fixed.
And yes, by VDE I mean Virtual Distributed Ethernet.
VDE is not part of the KVM project. The best way to deal with networking is to just setup bridges yourself.
I actually ended up giving up for a while on KVM entirely because the video device is horribly slow and VDE support is not reliable
How do you mean the video device is "horribly slow". Also, what do you mean by VDE (as in, vde.sf.net?).
2.6.20 will be the first real release of KVM. This benchmark used 2.6.20-rc3. For 2.6.20-rc4, a new shadow paging implementation was introduced (memory virtualization) that is significantly faster than what was present in -rc3. I've only got microbenchmarks handy, but context switch time, for instance, improved by about 300%.
I suspect if they reran their benchmarks with -rc4, the KVM numbers would be much more competitive with the Xen numbers (although I do suspect Xen will still be on top--slightly).
The best resource I know of is http://wiki.xensource.com/xenwiki/HVM_Compatible_P rocessors
Either you didn't pay attention or your school wasn't that good. Sorry.
A "console emulator" can be a straight forward emulator in which case, you should know enough from just your basic architecture courses. Did you discuss instruction decoding and ISAs?
Modern console emulators are probably Just In Time compilers. You should have had a compilers class and the prof should have at least mentioned binary translation. Even if they didn't, you should have spent a little time on JITs in an architecture class.
A college education is not necessarily about knowing how to solve problems, but how to decompose a problem into a series of problems that you can then figure out how to solve. For a console emulator, that may mean that first you know you have to read about the architecture your emulation (what's the ISA, what are the components, etc.). Then you realize you have to parse the actual ROMs (here's where your automata/compiler background kicks in). Then either emulate each instruction (tedious) or do dynamic translation.
If you wouldn't even know where to start, you didn't get the right education. I'd recommend trying to find a masters program or pick up some text books.
"While many users reading Slashdot embrace Windows, ZDNet is running an article on why the rest of the world isn't ready. One note for Windows developers: 'Stop assuming that everyone using Windows (or who wants to use Windows) is a Windows expert.' While a lot of these topics have been brought up as both stories and comments on Slashdot, this article pretty much sums up why Fedora Core 6 could be absolutely terrible, and people would still believe there is no other option."
If you are a Windows user (like the author of TFA), then you will be pissed off by Linux because you aren't as productive immediately (because, *gasp*, it's different).
If you are a Linux user (like me), then you will be pissed of by Windows because you aren't as productive immediately (because, *gasp*, it's different).
It's the same old story that's been told a thousand times before. Come on, it's not a slow news day. Hell just froze over.
Feel free to correct me if I've mis-understood, but your position seems to be that while it may be prohibitively hard to detect the a trojan hypervisor, it's still technically possible. Joanna presented as if BluePill were undetectable, you're contesting that it's detectable, just hard.
:-)
No, not really. I'm saying that in all circumstances, it's detectable. I'm claiming that even with all the engineers in the world working on this thing, it's still detectable (although it may be hard).
Practically speaking, building a VMM is really difficult. I have serious doubts that one could make even a reasonably complete version of this sort of thing. To make something that was more than trivially easy to detect would require an even larger under taking. And at that point, you'd have a VMM that was so good, I doubt you'd bother making malware. You'd start selling it for $800 a CPU and make yourself a fortune
An excellent idea, however if AV companies start doing this, what will BluePill++ start doing? Blocking NTP?
Maybe. But then, you use some other sort of protocol. It's just like the arms race between today's virii and anti-virus software. It's pretty much exactly what we've been dealing with for years.
Is this article talking about the paravirt_ops API that is being discussed on linux-kernel lately, or something
else?
It's just paravirt_ops.
If I understand your method correctly, it still requires the use of an external clock. If that's the case, there's still no programmatic way to do it, you're still relying on external sources, which is something your average end-user is unlikely to do.
.5 seconds, and we're claiming that under blue pill, rdmsr will take 10x longer, then you need run the rdmsr instruction in a tight loop for at least .05 seconds.
We're talking about anti-virus software here--not the average end-user. Anti-virus software just has to use NTP (or invent it's own time-keeping protocol) to check external time. There is of course a skew when checking time over the internet so you just make sure that your sample interval is much larger than your potential skew.
For instance, if you can only reliably get time from an NTP server plus or minus
Of course, you'd probably run it much longer just to be safe. For instance, if you ran it for 2 seconds, on native hardware it would take 2 seconds (and based on an external clock, you'd get anywhere from 1.5 to 2.5 seconds). Under blue pill, you would measure anywhere between 19.5 and 20.5 seconds. Such a dramatic difference would be a clear sign that you're in a VMM. That assumes that this is the ideal version of Blue Pill too which current doesn't exist. For less than ideal versions, there are much simplier tricks you can do.
She admitted that timing attacks were her weakness, as did the other guy who talked about virtualization-based rootkits. The problem is that you have to have a benchmark to compare it to, and you have to assume that the hypervisor doesn't modify the time whenever it is called.
The difference is order of magnitudes. Hypervisor exits incur penalties on the order of thousands of cycles. If you don't know the specifics of the system you're on, you can always compare the cost relative to a similiar instruction. For instance, while you may not know how many cycles a rdmsr should take, you should have a rough idea of the ratio of how many movl $0, %eax's you can do verses rdmsrs within a certain time period.
The hypervisor cannot modify external time sources because it cannot detect the use of an external time source in the general case.
The key to avoid skew is numbers. Do a ton of these operations and any potential skew will be lost in the noise.