Virtualizing Cuts Web App Performance 43%

czei writes "This just-released research report, Load Testing a Virtual Web Application, looks at the effects of virtualization on a typical ASP Web application, using VMWare on Linux to host a Windows OS and IIS web server. While virtualizing the server made it easier to manage, the number of users the virtualized Web app could handle dropped by 43%. The article also shows interesting graphs of how hyper-threading affected the performance of IIS." The report urges readers to take this research as a data point. No optimization was done on host or guest OS parameters.

Comments

Well,

[Fordiman]

Duh.

Seriously. I don't know who gave anyone the impression that virtualization was a performance booster. Management improver? Sure. Stability insurance? Why not? But if you don't get that virtualizing your servers imposes a bit of overhead, then you're probably not paying attention.

I especially love the idea that running different types of server virtualized on the same machine is a good idea; the idea of virtualization of multiple servers is to distribute the load. If you have, say, ftpd, httpd and mysqld running as their own virtualized systems, they will all get hit *simultaneously*.

Again. Duh.

Re:Well,

[Mr. Underbridge]

Seriously. I don't know who gave anyone the impression that virtualization was a performance booster. Management improver? Sure. Stability insurance? Why not? But if you don't get that virtualizing your servers imposes a bit of overhead, then you're probably not paying attention.

Well, I think the point was that he attached an actual number to the amount of the performance hit, which is relevant. That's called research; quantifying and proving that which seems 'obvious'.

Re:Well,

[hey!]

Well, it's not a surprise, but it's probably worth quantifying.

Here's a question: what is more available: hardware or skilled system administrators? Obviously hardware.

Here's a common scenario: you've set up a system to provide some useful package of services. How do you let other people duplicate your success? (1) tell them what hardware they need and (2) have them install and configurethe software on their hardware. Guess which item involves the most cost in the long run?

The hardware is easy; the greatest barrier and cost is the process of installing and configuring the software. That's one place a virtual machine is worth considering in producation systems. You aren't going to use something like VMWare in one-of-a-kind production systems. You're going to use it when you need to clone the same set up over and over again. This is very attractive for application vendors, who spend huge amounts of support on installation and tracking down compatibilty conflicts.

Another application would be an IT department that has to support dozens of more or less identical servers, especially if they are frequently called upon to set up new servers. If I had a choice, I'd use Linux virtualization on a midrange or mainframe, but if those servers must be Windows servers, then I'd be looking at some kind of cluster with SAN. This is not really my area of expertise, but we're talking high end boxen for x86; if the typical server didn't need 100% of box, then I have three choices: waste server bandwidth (expensive), force groups to share servers (awkward and inflexible; what if I have to separate two groups joined at the hip?), and virtualization.

Naturally if you are virtualizing production servers, you need to scale your hardware recommendation up to account for VM overhead.

What would be very interesting is a study of the bottlenecks. If you are considering a system with certain characteristics (processor/processors, memory, storage/raid etc) and you have X dollars, where is that best spent?

Re:Well,

[drinkypoo]

To us, the whole point of virtualization is that we have several servers which are mostly idle at all times and completely idle at most times, and for support reasons we're not supposed to be running anything else on the same copy of windows. So we can replace five or six systems with a four-core 1U box with just a few gigabytes of memory, which will recover rack space and reduce power consumption. For anything that's actually heavily loaded, running on the hardware is probably a very good idea.

Bogus Test

[Anonymous Coward]

Who uses VMWare Server in a production environment anyway? We run all of our Web services, Exchange servers and SQL databases in VMWare's Virtual Infrastructure 3. VMWare Player and Server are only ment for lab evironments and low load applications. VMWare even says as much on their website. Either this is just FUD or the author is an idiot. In other news water is wet.

Re:Bogus Test

[Sobrique]

Actually, the company I worked for 6 months back, one of the projects I was involved in was 'VMWare'. Production stuff running on on the ESX servers (which became 'virtual infrastructure') in our datacentre, as a cost effective scalable environment. Yes, we weren't getting 'uber performance' but then again, we were running 150 or so VMs on an 6 server VMWare farm.

One of the other things we prototyped and deployed was 'site services packages' - get GSX (now VMWare Server), stick it on a pair of 2U servers, and attach a storage array to both of them. Then create your 'template' fileserver, DHCP server, print server, proxy, that kind of thing and deploy them to this package. It worked very well indeed - you get a whole new order of magnitude on stability (although to be fair that's in part because we through away the crappy workstations that were doing the 'low intensity' stuff) and was extremely managable, and trivially replacable in the event of a hardware failure.

Performance? No, VMWare isn't that great on performance - whilst it's not bad, in an ideal situation, fundamentally what you are doing is introducing an overhead on your system. And probably contention too. But it's really good at efficient resource utilisation, easy manageability and maintainability.

As an experienced sysadmin, my reaction is screw performance. Let's start with reliable and scalable, and then performance just naturally follows, as does a really high grade service.

Proactive laziness is a fundamental of systems admin. Your job, is essentially to put yourself out of a job - or more specificially, free up your time to play with toys. The best way to do this is build something stable, well documented and easily maintainable. Then your day consists of interesting stuff, punctuated by the odd RTFM when something doesn't work quite right.

Re:Bogus Test

[sammy baby]

Yes: it performs much, much better.

VI3 is actually a suite of products. At the heart is VMware ESX Server, which is actually an operating system in its own right: it runs "on the metal," without having Windows or Linux installed already on the system. It also has a service console operating system which looks suspiciously like a *NIX style operating system, so you can SSH directly to the system, cd into your /vmfs directory and, say, scp disk files over the network. If you wanted to.

However, as a pretty damn safe rule of thumb, no system is going to run faster on equivalent hardware after being virtualized. In a prior job where I was often asked to provide development/test systems, I got phone calls from a lot of people who were bitten hard by the virtualization bug. Whenever someone brought up any issue having to do with infrastructure, no matter how odd or off the wall, they wanted to push virtualization as a solution. I had to explain to them that if your problem is that a web server is slow, the answer isn't to install VMWare server on it, set up two host operating systems, and say, "There! Now I have two web servers." You'd be surprised how pervasive that sort of thinking is, even among people who should patently know better.

Another useful guideline: various types of services are impacted differently by being virtualized. Generally, the best candidates for virtualization are ones that spend a lot of time idle. This is actually more common than you might think - people need a server set up for something, can't put it on a pre-existing system for security/compatibility reasons, so they go out and buy a new system which is ten times more powerful than they need. You can put a lot of these kinds of systems on a single, reasonably powerful ESX server. On the other hand, systems that heavily tax available hardware, especially I/O, are usually much harder to deal with.

Re:Bogus Test

[Bohiti]

Actually, as it's been explained to me, the ESX hypervisor itself is pure proprietary code (and small, too). The Service Console is very readily admitted to be a tweaked out RHEL (3, I believe..). Linux is used to boot, and then (magically?) transfers control of the bare metal to the hypervisor. Linux then jumps into a virtual machine, although it's not presented like a virtual machine, which creates all this confusion.

In the end, the tweaked RHEL that you interact with (ssh, scp) is not the hypervisor, but a VM with special tools that can manipulate the hypervisor.

Re:Virtualize this

[Fordiman]

I do like the idea of a variably sized beowulf cluster running a floating number of package (LAMP) servers. Get more clients? Add more VLAMPs. Things slowing down? Add more hardware.

You still take performance hits, but if you can scale your system by just adding cheap commodity systems, that works. Plug it in, boot it off a CD, and let the Cluster take control.

This is VMware Server and not ESX Server

[Fuyu]

They performed the test on VMware Server not VMware ESX Server which is what most enterprises will use. VMware ESX Server runs on "bare metal", so it does not have the overhead of the host operating system.

Bad data, bad setup

[duncanFrance]

There's quite a lot wrong with their setup.

1) As others have pointed out, they should be running on ESX to get best performance.
2) Physical machine was a dual-proc. How many processors did they assign to the VM?
3) Physical machine had 2GB memory. They assigned 2GB to the VM!! Vmware will take 256MB of this
for itself, so that 2GB visible to Windows will be being swapped.
4) How many disks did the physical machine have, and what was on them?
If e.g. the physical machine had two disks, the VM should have been given two disk files, with each file being placed on a different physical spindle.

You get the picture.

holy cow am I a nerd

[thegnu]

I do like the idea of a variably sized beowulf cluster running a floating number of package (LAMP) servers. Get more clients? Add more VLAMPs. Things slowing down? Add more hardware.

I started getting aroused as I read your post. This is highly disturbing.

"Duh!" moment

[Thumper_SVX]

I agree with many of the commentators here that this is pretty obvious. We use virtualization a lot, but also realize its limitations. For example, we don't run SQL or anything heavily transaction or I/O bound. CPU utilization is usually not a problem; virtual machines perform as well as their physical counterparts in most instances unless you have a lot of CPU intensive virtual machines running.

Web servers are mostly memory and CPU bound which would give one the impression that they would be great candidates for virtualization. However, VMWare Server is not the solution; network I/O is not good on Server. Typically your results would be maybe 75% of the actual physical speed on a "passthrough", less on a NAT. It depends a lot on how your network is set up, not to mention the abilities of the physical machine.

The best solution is Virtual Infrastructure (used to be ESX). That product tackles most of the failings of VMWare server and fixes them. The only exception is that I still wouldn't run anything I/O heavy on VI. SQL's a no-no. Also, if you're not getting the performance from a single web server that you expect, you can easily throw up more web servers. Now, obviously you might get into M$ licensing issues, but that's why you run your web services on Apache :D

Re:single data point is correct

[TheRaven64]

The biggest overhead from most forms of virtualisation is from emulated devices. If you have loads of money, you can give it to IBM and get some hardware with virtualisation-aware network and block device controllers. Then you get good performance. Alternatively, you can use paravirtualised device drivers. Xen supports this by default, and I think KVM does now for networks. Not sure about VMWare.

With paravirtualised devices, or devices that are virtualisation-aware, a VM can be within 10% of the performance of a real machine quite easily. Without I'm surprised they even got to 57% of native performance for web applications.