When VMware Performance Fails, Try BSD Jails

Siker writes in to tell us about the experience of email transfer service YippieMove, which ditched VMware and switched to FreeBSD jails. "We doubled the amount of memory per server, we quadrupled SQLite's internal buffers, we turned off SQLite auto-vacuuming, we turned off synchronization, we added more database indexes. We were confused. Certainly we had expected a performance difference between running our software in a VM compared to running on the metal, but that it could be as much as 10X was a wake-up call."

Comments

This is Ironic, right?

[Fahrvergnuugen]

Oh the irony

Safari canâ(TM)t open the page âoehttp://www.playingwithwire.com/2009/06/virtual-failure-yippiemove-switches-from-vmware-to-freebsd-jails/â because the server where this page is located isnâ(TM)t responding.

Re:This is Ironic, right?

[mvip]

We're working on it. The irony is that this is the only server that is still running as a VM (because it is a hosted VPS).

Re:This is Ironic, right?

[rachit]

Can we ask you then, why are you running it on VMware Server? Use ESXi. Its free. VMware Server's I/O performance is no where near as good as ESXi.

Re:This is Ironic, right?

[Anonymous Coward]

And also what hypervisor is under the VPS?

Re:This is Ironic, right?

[Anonymous Coward]

Then, it is NOT ironic.

Re:This is Ironic, right?

[hvm2hvm]

It's ironic from the OP's POV.

Re:This is Ironic, right?

[mvip]

Simple. It's a hosted VPS. We don't own or operate the core server.

Re:This is Ironic, right?

[mysidia]

Bad reason. Switch to a hosting provider with a clue, or help them get a clue. (Also, In modern versions of VMware Server, at least, the license specifically in section (9).1(b) forbids selling the use of a Virtual machine, and copies of VMware Server aren't sold for that purpose.)

From the VMware server EULA:

...provided such services may not consist of services to a third party that provide primarily computing or processing power (such as utility computing or grid computing) or any computer application-based service that is traded, rented, leased or sold on a Virtual Machine basis...

Anyways, this is just like other misconfigs a hosting provider can make. What would you do if they learned they were using RAID0 for your data storage, leaving their facility unlocked with no password security on the VM console, or you learned that their data center workers were regularly using the server's CD-ROM tray as a cup holder, and leaving pitchers of kool-aid on top of the server?

Just about any hypervisor, Xenserver, Hyper-V, etc, all blow the doors off (non-ESX) VMware Server.

And running on brand new, highly beefy hardware won't save you, in fact, the peformance penalty gets bigger; VMware Server does not scale.

Re:This is Ironic, right?

[MyDixieWrecked]

Use ESXi. Its free.

Since ESXi became free, I've installed it on several servers at work. The problem is that its hardware requirements are pretty specific. It won't install on just any PC. It would be nice if I could install it on some of the older servers we have kicking around (DL-140s) or some decommissioned desktops, but it just doesn't support those pieces of hardware.

The primary reason that Server's disk I/O performance is so horrid is that your VM's disk(s) is being stored as a file on the host OS's filesystem. That causes extra layers of system calls to access files in the guest OS. Between talking to the virtual disk that has to get translated to physical disk, plus the guest OS's filesystem, which appears to the guest as a contiguous physical disk, may become fragmented on the host OS; double fragmentation can occur which causes SERIOUS performance penalties. ESX has a specialized vmfs that it uses to store your images which is designed for VM performance.

Personally, I would recommend Xen over ESX if you don't have the proper hardware and/or don't want to pay licensing fees. Although it's got a higher learning curve, it's easier to automate (especially since ESXi got rid of CLI support) and there are a plethora of free tools and documentation around. Being that ESX isn't free, it's harder to find support in forums other than VMWare's own site.

Also, VMWare Server's performance in general leaves a lot to be desired. I would *never* use it for production systems. I've had it installed on machines with 6x15K SaS drives (this is before ESXi) and 8 cores and it would start to choke after about 4 or 5 VMs. Plus VMWare Server doesn't handle multi-core VMs very well. Incredible performance issues arise and you're better off creating your application to scale out to multiple single-core VMs rather than make them dual-core. Server also doesn't handle memory nearly as well as the Type1 hypervisors that Xen and ESX use.

Re:This is Ironic, right?

[hesaigo999ca]

I would guess to that if they don't know about this, they also don't know there are certain things you just don't virtualize either, like a server that has an SQL server on it...with big logs with lots of read and writes. There is something to be said about when to use VMWare, and when not.

Re:This is Ironic, right?

[Mysticalfruit]

I hear ya...

We've deployed a number of ESXi boxes with mixed results. In a nutshell, if you have a underutilized server that's lightly loaded, it's a great candidate to be a VM.

Anything that needs performance should be considered off the list for a VM, unless you can convince the consumers that the speed penalty is worth the ease of management.

* Web server that servers your lunch menu and maybe your HR vacation scheduling system: VM
* Build machine that pounds through 7 GB of source and takes 10hrs of solid compiling to produce output: not a good choice for a VM.
* Domain Controller: VM

I like VM's because they're portable and machine agnostic. The whole virtual infrastructure stuff is polished and works.

However, people tend to think that virtualization is the right path for everthing, which it is not.

It has its place in the IT toolbox with everything else. Who knows, maybe Intel and vmware will hatch an offspring that looks like ESXi but with hardware provisioning... Though I'm sure IBM's lawyers would shit kittens!

Re:This is Ironic, right?

[AVee]

Or you happily virtualize it but use an actual physical partition to as the 'virtual' harddrive. I'm not entirely sure VMWare server supports this, but Workstation does, and so does VirtualBox. It saves you one filesystem layer (then add a DB able to use raw partitions and lose the other filesystem layer as well).

Re:This is Ironic, right?

[CAIMLAS]

The filesystem of choice - both in vmware as well as on the host server - makes a huge difference in the disk i/o performance (in Linux). The long and the short of it is that ext3 has become a pretty atrocious mess wrt disk IO, and you'd be better served with something like XFS. I'd wager a guess that in substituting 'xfs' for 'ext3' in ext3 -> kernel -> vmware -> ext3 -> kernel you'd shave at least 30% off the time required to do things.

Re:This is Ironic, right?

[Anonymous Coward]

"if you have a underutilized server that's lightly loaded, it's a great candidate to be a VM.
Anything that needs performance should be considered off the list for a VM"

In other news, mixing hot and cold water gives you warm water. News at eleven.

excellent sales story

[OrangeTide]

Virtualization is an excellent story to sell. It is a process that can be applied to a wide range of problems.

When applied to a problem it seems to create more performance issues than it solves. But it can make managing lots of services easier. I think that's the primary goal to these VMware-like products.

Things like Xen take a different approach and seem to have better performance for I/O intensive applications. But a Xen hypervisor VM is in some ways more similar to a BSD jail than it is to VMware's monitor.

VMware is more like how the Mainframe world has been slicing up mainframes into little bits to provide highly isolated applications for various services. VMware has not caught up to the capabilities and scalability to things IBM has been offering for decades though. Even though the raw CPU performance of a PC is better than a mid-range mainframe at 1% of the cost (or less). But scalability and performance are two separate things, even though we would like both.

Re:excellent sales story

[gfody]

Most of the performance issues and I think also the issue faced in TFA have to do with IO performance when using virtual hard drives especially of the sparse-file, auto-growing variety. If they would configure their VMs to have direct access to a dedicated volume they would probably get their 10x performance back in DB applications.

It would be nice to see some sort of virtual SAN integrated into the VMs

Re:excellent sales story

[Anonymous Coward]

Virtualization is an excellent story to sell. It is a process that can be applied to a wide range of problems.

Virtualization makes many things a lot easier. Testing, rollback, provisioning, portability & backup.

The success of virtualization is due to failures of the software industry to have good separation between applications & operating systems. The one-application-per-server trend is the result, which leads to a lot of idle capacity.

Re:excellent sales story

[Eil]

But a Xen hypervisor VM is in some ways more similar to a BSD jail than it is to VMware's monitor.

Actually, Xen is not at all similar to a BSD jail, no matter how you look at it. Xen does full OS virtualization from the kernel and drivers on down to userland. A FreeBSD is basically chroot on steroids. The "virtualized" processes run exactly the same as "native" ones, they just have some restrictions on their system calls, that's all.

I guess the thing that bugged me about the most about TFA was the fact that they were using VMWare Server and actually expecting to get decent performance out of it. Somebody should have gotten fired for that. VMWare server is great for a number of things, but performance certainly isn't one of them. If they wanted to go with VMWare, they should have shelled out for ESX in the beginning instead of continually trying to go the cheap route.

Re:excellent sales story

[ckaminski]

What are you talking about? ESX has supported REAL SANS since almost day one. I've been able to GREAT things on a single vmware server, in one instance I managed 25 2GB J2EE app VMs on a quad core XEON (2005 era). In another I managed 168 sparsely used testing VMs (2x quad core). But I've ALWAYS had trouble with databases and Citrix, in particular, with VMware.

Storage is only part of the issue. Having to run 10-160 schedulers *IS* the issue. Vmware doesn't utilize efficiencies in this arena like Xen or Jails, or OpenVZ or Solaris Containers can.

Re:excellent sales story

[QuoteMstr]

First of all, you have to admit that the product line names are confusing. You'd expect a product with the word "server" in its title to be useful for, well, servers. Second, even ESX is still less efficient than just using a kernel to isolate different processes. That's what it's there for, after all.

Re:excellent sales story

[jgtg32a]

Your sig isn't logically sound just because the Jews win because you lost doesn't mean you win when the Jews loose.
Just something I thought I'd point out.

On another side note what happened to the fine art of trolling, people these days just throw a bunch of racial slurs together and think they're all that. In my day it took a certain finesse to troll properly, you had to be well informed on the issue and then speak truths on the issue and then interpret those truths in a way that will set people off.

oh, well
get off of my lawn

Re:excellent sales story

[syousef]

Virtualization is an excellent story to sell. It is a process that can be applied to a wide range of problems.

Screw-drivers are an excellent tool. However if you're in a position to buy tools for your company, you should know enough to show me the door if I try to sell you a screw driver to shovel dirt.

Right tool. Right job.

In any industry:

Poor management + slick marketing = Disaster

Re:excellent sales story

[MichaelSmith]

I really should have kept a copy of those "don't feed the trolls" ascii art pictures people used to post on usenet. It would have come in handy here.

Re:excellent sales story

[mysidia]

Totally unnecessary. If you want a 'virtual SAN', you can of course create one using various techniques. The author's biggest problem is he's running VMware Server 1, probably on top of Windows, and then tried VMware Server 1 on top of Ubuntu.

Running one OS on top of another full-blown OS, with several layers of filesystem virtualization, no wonder it's slow; a hypervisor like ESX would be more appropriate.

VMware Server is great for small-scale implementation and testing. VMware server is NOT suitable for mid to large-scale production grade consolidation loads.

ESX or ESXi is VMware's solution for such loads. And by the way, a free standalone license for ESXi is available, just like a free license is available for running standalone VMware server.

And the I/O performance is near-native. With ESX4, on platforms that support I/O virtualization , Vt-d/IOMMU, in fact, the virtualization is hardware-assisted.

The VMware environment should be designed and configured by someone who is familiar with the technology. A simple configuration error can totally screw your performance. In VMware Server, you really need to disable memory overcommit and shut off page trimming, or you'll be sorry -- and there are definitely other aspects of VMware server that make it not suitable at all (at least by default) for anything large scale.

It's more than "how much memory and CPU" you have. Other considerations also matter, many of them are the same considerations for all server workloads... e.g. how many drive spindles do you have at what access latency, what's your total IOPs?

In my humble opinion, someone who would want to apply a production load on VMware server (instead of ESX) is not suitable briefed on the technology, doesn't understand how piss-poor VMware server's I/O performance is compared to ESXi, or just didn't bother to read all the documentation and other materials freely available.

Virtualization isn't a magic pill that lets you avoid properly understand the technology you're deploying, make bad decisions, and still always get good results.

You get FreeBSD jails up and running, but you basically need to be skilled at FreeBSD, and understand how to properly deploy that OS in order to do it.

Otherwise, your jails might not work correctly, and someone else could conclude that FreeBSD jails suck, stick with OpenVZ VPSes or Solaris logical domains.

Re:excellent sales story

[gfody]

If you have a real SAN sure, IO performance is probably not your problem. If not then you might just try to use a sparse-file virtual hard disk and experience incredibly bad IO performance. My experience is really only with VirtualBox where the virtual disk is the only thing available in the UI and setting up direct disk access is advanced, text-based config - I'm not sure if it's like this with ESX - but I think it would be nice if instead of that whole virtual hard disk crap if the VM host was also a SAN server for your VMs and you just always use SAN.

Re:excellent sales story

[Thundersnatch]

It would be nice to see some sort of virtual SAN integrated into the VMs

Something like this you mean? Turns the local storage on any VMware host into part of a full-featured, clustered, iSCSI SAN. Not cheap though (about $2500 per TB)

Re:excellent sales story

[Feyr]

seconded. last time i tried, vmware server couldn't handle a single instance of a lightly loaded db server. moving to esx we're running 6 VM on that same hardware and the initial server has near-native performances

in short. use the right tool for the right job, or you have no right to complain

Re:excellent sales story

[masshuu]

this?
+----------+
|  PLEASE  |
|  DO NOT  |
| FEED THE |
|  TROLLS  |
+----------+
    |  |
    |  |
  .\|.||/..

Re:excellent sales story

[MichaelSmith]

That would do.

Re:excellent sales story

[debatem1]

The big thing here is that a lot of businesses conflate *emulation* and *isolation*. Virtualization is for when you want to emulate the behavior of one system on another. Containers are for when you want to isolate a process or a set of processes from screwing with everybody else's stuff. Don't use one when what you need is the other, and everybody walks away happier.

Re:excellent sales story

[aarggh]

In my opinion it always comes down to the fact that shelling out some money for a good product always beats trying to stuff around with a "free" one that's hard to configure and maintain. I run 4 ESX farms, and have NO problem rolling out virtually any type of server from Oracle/RHEL, to Win2k3/2k8, and everything inbetween. I simply make sure I allocate enough resources, and NEVER over commit. I did a cost analysis ages back trying to convince management we needed to go down the virtualisation path to guarantee business continuity.

In the end it took the failure of our most critical CRM server crashing and me importing an Acronis backup of it into ESX that convinced them beyond a shadow of a doubt.

I would say to anyone, something for $15-20K that gives:

Fault-tolerance
Fail-over
Easy server roll-outs
Simple network re-configuration
Almost instant recoverability of machines

Is more than worth the cost! The true cost of NOT doing it can be the end of a business, or as I have seen, several days of data/productivity lost!

Performance issues? Reliability issues? I have none at all, the only times i've had issues are poorly developed .NET apps, IIS, etc, which I then dump the stats and give them to the developers to get them to clean up their own code. And more than once I've had to restore an entire server because someones scripts deleted or screwed entire data structures, and in a case like that, being able to restore a 120GB virtual in around 30mins from the comfort of my desk or home really beats locating tapes, cataloging them, restoring, etc, etc.

I have Fibre SAN's (with a mix of F/C, SAS, and SATA disks) and switches, so the SAN just shrugs off any attempt to I/O bind it! The only limitation I can think of is the 4 virtual NIC's, it would be good for some of our products to be able to provide a much higher number.

No comparison in my opinion.

Re:excellent sales story

[mysidia]

ESXi is free, and they could have used that. The overhead for most I/O is a fraction that of VMware server's.

If they did this so long ago that ESXi wasn't available for free, then their basis for discussing problems with VMware is way outdated too, a lot changes in 14 months....

VMware Server simply has many issues: layering the VM filesystem on top of a bulky host filesystem. Relying on a general purpose os to schedule VM execution, memory fragmentation, slow memory ops, contention for memory and disk (VS inappropriate host OS caching/swapping), etc.

And it bears repeating: Virtualization is not a magic pill.

You can't deploy the technology and have it just work. You have to understand the technology, make good design decisions starting at the lowest level (your hardware, your network, storage design, etc), configure, and deploy it properly.

It's not incredibly hard to deploy virtualization properly, but it still takes expertise, and it's not going to work correctly if you don't do it right.

Your FreeBSD jail mail server might not work that well either, if you chose a notoriously-inefficient MTA written in Java that only runs on top of XWindows.

Re:excellent sales story

[DaemonDazz]

Actually, Xen is not at all similar to a BSD jail, no matter how you look at it. Xen does full OS virtualization from the kernel and drivers on down to userland. A FreeBSD is basically chroot on steroids. The "virtualized" processes run exactly the same as "native" ones, they just have some restrictions on their system calls, that's all.

Precisely.

Similar products in the Linux space are Linux Vserver (which I use) and OpenVZ.

Re:excellent sales story

[Night64]

Actually, Xen does paravirtualization very well. We use a flavor of it (a major enterprise one, but I'm not paid to tell names) with great success in a production environment. On our environment at least, it has a better performance than ESX in the same hardware. We don't use Windows servers very much, but this flavor (hint, hint) works very well with Windows in a paravirtualized setup. A little better than ESX, but, as always, your mileage may vary.

Re:excellent sales story

[Anonymous Coward]

Slightly off-topic, but I'm moving off Windows from my Lenovo T61 laptop but I'd still like the option of having a virtualized version of it on Ubuntu 9.04. What do you guys think of vmware, virtualbox, and the others? I've read a couple comparisons online (informal and formal) but haven't really made up my mind.

Re:excellent sales story

[ardle]

When you say "25 2GB J2EE app VMs", are you saying that each app got its own VM (no J2EE clustering)? Or was it a sort of virtual server farm with applications deployed across multiple virtual servers?
I'm trying to get a picture of how virtualisation gets used...

Re:excellent sales story

[BaldingByMicrosoft]

TFA wasn't running ESXi? Thanks, now I can skip the read entirely. Silly TFA.

Anyway, isn't "virtualization" so last year? "Local cloud" is the groove.

Re:excellent sales story

[mlts]

Virtualization isn't just about performance:

If you have physical machines connected to the same SAN, both VMWare's products and Microsoft's Hyper-V support running failover clustering. This way, if one of the machines goes down, the VM and its services keep running with perhaps a small delay (in milliseconds) while the handoff to the other machine takes place.

The advantage of failover clustering at this level as opposed to the application level is that not all applications have the ability to use clustering, and a lot of companies may have database utilities that support clustering, but the app running on it may have issues, and making code for a handover at the app level might be impossible, especially if the utility is a niche item.

Another advantage of virtualization is the ability to do a hardware upgrade without affecting anything in the VM. For example, I have had a VM running on my Linux box that was a DHCP server and DNS cache. When it became time to move to a new machine because the 1999 vintage Linux box used too much power, I just copied the VM disk files to the new box, upgraded the client side OS drivers, and called it done. To stuff running in the VM, all it would notice is that it run faster, but everything else would be exactly the same. Similar with filesystem manipulation. If I want to move the VM to a new disk, I just turn it off, move the files to the new volume, turn it back on. The VM doesn't care one bit that its virtual disks are on a new SATA drive than an old IDE.

OS snapshotting comes to mind too. One of the uses I use a VM for is snapshots and rollbacks. For example, I tend to browse the Web in a VM under VirtualPC. This way, if some malicious code makes it past Firefox/Adblock/NoScript, it will only affect the VM (barring a hole that allows code to affect the hypervisor executable), and a simple click on the close window button dumps all changes. Another use is system upgrades. Say I do a major upgrade or patch of an application and the app won't start. I can rollback to a previous snapshot, turn the VM on, and be able to resume production with that machine without running out of time in the service window.

Filesystem manipulation. If you have the software, backing up VMs becomes easy. You have either a tape drive, a disk to hold stuff on, or both. The VM can be happily running, and at the same time, its filesystem can be snapshotted outside the OS and backed up.

There is a penalty for using a VM, and that is performance. However, this is mitigated significantly by having drivers in the client OS. For example, Hyper-V has a very fast virtual network switch, and it supports a virtual 100baseT adapter. So, just by installing the client drivers, a VM can communicate with others on the same virtual switch a lot faster than without.

Another penalty is unknown security issues. There is always the concern that a hypervisor can be compromised, and malicious code that is running in a VM can get the machine access of the hypervisor (whose drivers for a lot of tasks might be running with root or kernel level authority). This can be mitigated by making sure the guest operating systems are secure.

Re:excellent sales story

[micheas]

I use libvirt virtual-machine manger, kvm (kernel virtual machine) and qemu for my desktop. It works well for my uses for windows 2K, windows xp, windows 7, FreeBSD, Debian and Cent OS virtual machines.

Sound took a little to get working and windows 7 is missing the aero effects because the virtualized graphics card sucks.

Over all as long as I run fewer virtual machines than cpu cores my desktop stay's pretty responsive,

The only problem I've had was windows update is running on win2k, XP and 7 at the same time, file system IO started to make every thing fairly painful.

Over all it is really nice to be able to look at IE6, 7, and 8 at the same time while debugging a website, which is most of what I do in windows.

I haven't really stressed the virtual machines much, I use Linux because I have more experience with UNIX type operating systems than I care to admit. So, If I have a choice I use Linux, but for when I need ot use Windows, the libvirt, kvm, qemu stack has been really nice.

VMware has been more or less unusable for testing javascript and flash animations. It is fine for servers, but when you launch a web browser, the experience has always been painful for me.

Re:excellent sales story

[BitZtream]

Wow, how about you make it more obvious that you have no clue what you're talking about.

ESX and ESXi are bare metal hypervisors that run on the hardware directly. They do not require any OS.

Management of the system can be done with the VMWare infrastructure client GUI which runs on Windows.

The management interface is a SOAP service and the API is public, you can admin via a perl script if you want, and indeed VMware has made command line tools (written in perl IIRC) that access the soap interface. These tools are all available as pre-packaged virtual machines if you want, based on Linux VMs and can be downloaded directly from the web server on the ESX or ESXi server.

Now if you want to bitch about the fact that I can't use FBSD as a host for a virtual machine then by all means, but your complaints are just those of ignorance.

I've used FreeBSD since 2.2, and I'm guessing from your post that you're one of those people that still tries to use FreeBSD as a desktop machine. While it obviously can be done, and with enough effort it can be rather usable, FreeBSD really isn't intended for the desktop PC role, you may want to consider using an OS more suited for the task, let FreeBSD remain the bad ass server that it is and let OS X and Windows be the desktop OSes that they are.

Re:excellent sales story

[BitZtream]

How long have you worked at VMWare?

Your opinion is a sales pitch, thats about it. If the only 'issues' you've had are with '.NET apps and IIS' it leads me to believe you're also just a Linux zealot.

You're impressed/proud of the fact that you can do a restore via VMware from your desk? Thats not impressive, I INSTALLED ESXi on a machine in our data center (20 miles away) from my desk, its not impressive then either. Real hardware can do all this without VMware, and has done so for years, have you never worked with real server hardware?

You're all giggly talking about your Fibre SANs with a mix of 'Fibre/Channel', SAS and SATA. You you have fibre channel connected fibre channel? You're worried about having only 4 virtual NICs? Why exactly? Its not like they are limited to 1 gig of bandwidth each, they are virtual, they'll happily take and send more internally. Not enough NICs for your wild network segmentation? Welcome to VLANs.

Seriously, how long have you worked for VMware? Whatever it is, I suggest you learn more about what you're talking about before trying to come to slashdot and brag about how cool your coolaid is.

Re:excellent sales story

[mysidia]

It's a lot harder and not particularly advisable, but it's completely possible to manage a standalone host without any Windows machine.

It just happens the most useful management tools that ship with the product are for Windows.

You can use a command line interface to start/stop VMS, setup datastores, etc, create a VM.

As for gaining access to the console, if you drop the right lines into the .VMX file before starting the VM, you can direct the host to accept a VNC connection on a port you designate, to access the VM console.

But time is precious, and in most enterprises it's most cost-effective to just round up a couple of Windows workstations and install the VI client on them to manage the VM hosts.

Re:excellent sales story

[rachit]

The only limitation I can think of is the 4 virtual NIC's, it would be good for some of our products to be able to provide a much higher number.

ESX 4 (very recently released) supports 10 NICs.

Re:excellent sales story

[Anonymous Coward]

From the story:

Certainly we had expected a performance difference between running our software in a VM compared to running on the metal, but that it could be as much as 10X was a wake-up call.

Oh look, if I take the anvil off my head it stops hurting. Anvils really suck!

When applied to a problem it seems to create more performance issues than it solves.

Who turns to virtualization for performance? Credible claims about VMs are made regarding consolidation/utilization, manageability and redundancy but better performance? I recall some recent interesting results from VMware showing a cluster of Oracle VMs outperforming Oracle's own RAC, but I think that says more about RAC than it does about VMware.

Have you got dozens of crufty old boxen running a collection of crufty old "Mission Critical (tm)" applications? Virtualization could make a lot of sense. Do you spend a lot of labor erecting platforms that tend to expire fairly quickly as focus moves to the Next Big Thing? Again, virtualization is your friend. Are you attempting to satisfy an SLA with N+1 or better redundancy? Virtualization might be a good way to get there.

Are you trying to squeeze the maximum pages per second out of however much hardware you can afford? Why would someone think net capacity of a given amount of hardware would be increased by adding virtualization overhead?

That had to be some PHB getting sold over martinis by VMware sales droids.

Re:excellent sales story

[OrangeTide]

I disagree. I consider Xen to be a kernel which other kernels are modified to run inside of, it is just a guest kernel making requests(read system calls) to a hypervisor(a special sort of kernel) that then translates it into requests to the host kernel. But mostly I feel this way because of the way I/O is handled in Xen is very much unlike the way VMware does it (go find my resume, I used to be an ESX developer at VMware).

Because Xen was originally designed to function without special hardware extensions to support virtualization it is a virtual machine in the same sense that Unix is a virtual machine(processes were literally virtual machines from day 1 in Unix). Xen just jams one more layer above processes.

BSD Jails are just a more Unix way of virtualizing a set of processes than Xen is. Xen requires an entire kernel to encapsulate the virtualization, BSD jails do not. In my opinion that is where they differ the most, but that difference is almost unimportant.

Re:excellent sales story

[Spad]

http://communities.vmware.com/community/beta/vcserver_linux

Beta, but available and you can run it in a VM on top of ESX so you don't have any additional costs other than the hit of running one extra (fairly low impact) guest OS in your ESX environment.

Re:excellent sales story

[moon3]

Virtualization came to life not to solve developer's voes, but to enable firms to sqeeze even more servers in a single rack (this is where the money is made on VM). You can run 8 or more virtual servers on one metal. The performance is terrible, of course, but developers hardly even use those resources, most the these servers are idling. Hosting companies are very happy with the results.

Re:excellent sales story

[aarggh]

I would actually say that the day ESXi became free, it made server completely obsolete for ANYTHING other than initial testing or building.

As you stated, this article really on every level is a ridicuously poorly designed implimentation, I don't get into flame wars as to what's the better OS, etc, etc, so far as I'm concerned whatever is best at doing what I need it to is the solution I aim for, and with ESX I must admit I have been extremely happy with the time and resource savings, as well as the GREATLY reduced management overhead. Throw in the HA, DRS, vMotion, and disaster recovery, and I now sleep a lot better at night, and get far fewer calls!

Re:excellent sales story

[headLITE]

In addition to virtualisation being an excellent story to sell, VMware has done a great job on marketing its product as *the* virtualization solution. I guess it's expected that people will now try VMware first before considering their actual requirements. FreeBSD jails, or Solaris zones, are virtualisation technologies too, they're just not as present as VMware is. And chroot is one too, even though nobody ever thinks of that anymore thanks to VMware's marketing. They can't do all that VMware can do, but most likely, if you don't actually intend to consolidate multiple different platforms on one kind of server, or you don't actually need your managers to be able to move servers around, you don't need VMware. And if all you want to do is isolate individual *server processes* that otherwise run on the same platform, then VMware is most definitely not what you need. Basically, everything starting with plain old chroot could potentially do a better job unless you have very specific needs that can only be fulfilled with complete virtualisation.

Re:excellent sales story

[aarggh]

Your an idiot! Sorry but I can't see any other way to put it in terms that you might actually understand.

Clearly you have NO idea what MOST IT departments or REAL data centres run like. You know, the ones in the real world that were built by an ever changing bunch of people and developers who float between departments and jobs, never adequately document anything they do, create all manner of bizarre work arounds and tricks to either make up for OS or design limitations or bad design, never budget or even PLAN for the (cough, cough!) EXTREMELY UNLIKELY scenario that the server might actually fail, and when it does and the people who actually know how it works, or even know how to properly do application restores, aren't around, and your getting the call at the footy on Saturday arvo to "GET IT WORKING".

You can bitch and moan all you want about processes and procedures, backup and restore documents, but in the many years I've been working, I have invariably found that if the group responsible for deploying a service, can offload that onto the IT dept, that's exactly what happens. And usually with NO EXTRA BUDGET to cover things like, you know, tape units, tapes, etc, etc.

So if my positive experience using ESX sounds like a sales pitch, then hell yeah! It has saved me enormous amounts of head-aches, time, etc, while providing complete fault-tolerance. You know, the kind of things most departments EXPECT from the IT guys but NEVER actually budget for! Using it I can now run and manage literally hundreds of servers with ease, compared to constantly chasing my tail looking after a fraction of that in out-dated and unsupported physicals that no-one wants to rebuild or replace!

Moron!

Re:excellent sales story

[raju1kabir]

Management of the system can be done with the VMWare infrastructure client GUI which runs on Windows.

This is one of the many reasons I gave up on VMWare - the management tools are primitive in that they only really work via the GUI and can't easily be scripted or accessed in an efficient manner. Trying to do significant management via the CLI was nigh unto impossible.

For people who have a Windows machine and excellent connectivity to the server room at all times, maybe that's okay.

For someone who is frequently out and about, it's not - using a prepaid SIM card on a Moroccan train I can easily reconfigure my Xen servers in the USA. With VMWare I haven't a prayer.

Re:excellent sales story

[Anonymous Coward]

I dig PC-BSD (FreeBSD for the desktop). Its almost as good as the latest Ubuntu, but admittedly a tad worse. Its made great progress over the last few releases so I'm hoping that pretty soon it will be equivilant to Linux's offerings.

Re:excellent sales story

[mrmeval]

I'd wondered why Virtualbox dragged ass. I didn't look to see if it would do direct access but now I will.

Re:excellent sales story

[Bert64]

More so in the windows world than unix...

I have always run a large number of services on a single unix system, sometimes splitting them up through the use of chroot, always isolating them from each other by running as different users... The only benefit i see from virtualization would be having each machine as a simple container that can be moved around different physical hardware, but then again copying a full linux install from one disk to another is not that hard, and unless you have a heavily customized kernel it should boot up just fine on a different machine.

Having individual apps isolated, so they could be moved to dedicated machines if the load increased would be useful, but chroot could buy that, on the other hand nothing i'm doing right now stresses 5 year old hardware so i could move the whole install to a newer more powerful machine.

Re:excellent sales story

[gbjbaanb]

they're not really bare metal hypervisors - they're cut-down linux OSes. ESX uses a redhat-based linux, ESXi uses busybox IIRC. VMWare Server (as faras I'm concerned) is a user-level hypervisor that runs on Linux too - or at least that's how I always install it :)

You can configure ESXi to open a SSH console up, but it has few commands and is unsupported. People generally use that to script VMs off to a backup server using SFTP (no rsync :( )

Re:excellent sales story

[leuk_he]

But you should not overlook the fact that NO virtulisation beats the performance of a virtulisation solution in 9 out of 10 times. If you load your servers already to 100% virtulisation will only add load.

If you have multiple lightly loaded servers you can consolidate those in a virtulized solution and safe money.

IF you have some server that are under full load you do not want to add anything that adds load.

jail is a solution, but the fact that they did major apllicaiotn changes it could also be done in the application instead, ginving each run a seperate configuration set.

Re:excellent sales story

[Anonymous Coward]

This way, if some malicious code makes it past Firefox/Adblock/NoScript, it will only affect the VM (barring a hole that allows code to affect the hypervisor executable)

and this is different from chroot/jail how?

Filesystem manipulation. If you have the software, backing up VMs becomes easy

And this is different from LVM/zfs how?

I agree that for Virtualisation is great for managing machines (failover,migration,etc) but running a whole os in a vm just for browsing is ridiculous, it offers no real advantage to a chroot/jail while giving you more things that need to be kept up to date and so is infact *slightly* less secure

Re:excellent sales story

[Psiren]

they're not really bare metal hypervisors - they're cut-down linux OSes. ESX uses a redhat-based linux, ESXi uses busybox IIRC. VMWare Server (as faras I'm concerned) is a user-level hypervisor that runs on Linux too - or at least that's how I always install it :)

Not strictly true. The service console is a cut-down Linux. The actual hypervisor is VMWare's own creation.

Re:excellent sales story

[RivieraKid]

Actually, they are bare-metal hypervisors. In the latest versions and probably the last couple of versions too, the Linux management console you speak of is in fact a VM running on the ESX hypervisor.

Re:excellent sales story

[chthon]

I think this is a really nice summary of most of the computer industry.

Re:excellent sales story

[Tony Hoyle]

The problem with Xen is its I/O perfomance sucks donkey balls - all I/O is performed in software by a qemu process, and on the mailing lists they recommend you pin that to a processor to improve performance... but that's a poor hack, and doesn't really work.

VMWare server may not scale but it's ten times better than Xen for this. ESX or ESXi beat it sideways.

Re:excellent sales story

[Anonymous Coward]

We are ALL impressed with your USE of BOLD even though you NO CLUE what you are TALKING ABOUT.

Re:excellent sales story

[AigariusDebian]

Or they could use an actual database. One that is designed for performance. Like MySQL or even Oracle if they are so big. SQLite is not suited for any production deployment. It is a good database for development and, possibly, embedded work, but for anything bigger than an iPhone app you should use a real database in production.

Re:excellent sales story

[dhfoo]

That and the requirement for Oracle as the DBMS. Hahahahahahahahhahahah.

Re:excellent sales story

[Znork]

Interesting product, hadn't seen it before.

I think you can accomplish pretty much the same thing with DRBD synced volumes shared over iSCSI, if you want it for cheap tho.

Re:excellent sales story

[Anonymous Coward]

Maybe you should be in the office working instead of gallivanting around the world playing around.

Re:excellent sales story

[Junta]

Windows *is* required for many ESX/ESXi environments. Specifically, if you want many of the features, you must run VirtualCenter, which requires a Windows server. Live migration is a feature they currently tie to that product and *don't* expose via a public API straight to an ESX(i) hypervisor.

In terms of the 'perils' of a full blown OS 'over' another OS, that may not be as big of a deal. Xen and VMWare ESX have similar strategies of a management OS that runs as a privileged guest, true. They feel they best serve by being in full control. However, if configured right, Linux hosted virtualization guests, for example, can acheive very good IO with 'normal' looking guest devices. They can achieve great IO performance with paravirtualized adapters. A 'full-blown' os can be just as optimized to fulfill the role as hypervisor fine, there is no theoretical reason why not.

In terms of dismissal of FreeBSD as a desktop platform, I think that unwise. I personally don't do FreeBSD, but I do use Linux and I achieve great personal benefit.

Re:excellent sales story

[raju1kabir]

Maybe you should be in the office working instead of gallivanting around the world playing around.

Maybe my job requires it.

Re:excellent sales story

[Znork]

I used to use ESX, but the built in virtualization in RHEL does it better these days. ESX performance is nice enough, but paravirt xen tech outperforms it by 3x on some things (scripts, exec, syscall intensive stuff).

It's also much, much cheaper.

Then again, I don't run any virtualized Windows, so your mileage may vary.

Re:excellent sales story

[maxwell demon]

How did you get that through the junk character filter?

Re:excellent sales story

[Thundersnatch]

DRBD + LVM or another roll-it-yourself solution wouldn't compare from a High-Availability standpoint; that would be an active/passive solution. LeftHand gear (now owned by HP) is a true N+1 or N+2 cluster. All nodes are active, and data is striped amongst all nodes with one or two copies. Lose a node, and iSCSI redirects keep everything working without any interruption at all. We actually use a cluster of five of their hardware appliances (basically HP DL320s with 12 disks each running the software on the metal) for our VMware installation, and are testing the Virtual SAN Appliance for our DR site.

Other "real SAN" features include very space-efficent snapshots without reservations (LVM is pretty useless for snapshots because of the reservations), volume clones with shared blocks, asynchronous replication with bandwidth controls, thin provisioning, grow/shrink, etc. If they could figure out a way to do truly clustered NFS/CIFS with deduplication on the same platform, they could kill NetApp in a year or two. HP bought LeftHand as a hedge against Dell/EqualLogic, just in case the "clustered storage" thing took off. It might still.

Re:excellent sales story

[maxwell demon]

I agree that for Virtualisation is great for managing machines (failover,migration,etc) but running a whole os in a vm just for browsing is ridiculous, it offers no real advantage to a chroot/jail while giving you more things that need to be kept up to date and so is infact *slightly* less secure

I'd already consider chroot just for browsing ridiculous. If you are paranoid, just run the browser under a different user ID.

Re:excellent sales story

[Anonymous Coward]

The article doesn't mention, that I found, what the OS with VMWare was - if linux, there is a longstanding NFS bug in linux which resolves by setting the NFS to run over TCP rather than UDP. I've had to argue many times with engineers who are stuck in UDP always providing better performance than TCP for NFS - sure, but not for linux. Sniff the NFS traffic and watch the lock up as retransmits take over the stream.

Not saying that VMWare is not part of the performance issue - just that I find the SQL over NFS, if the OS was linux, very suspicious.

Re:excellent sales story

[Night64]

The problem with Xen is its I/O perfomance sucks donkey balls - all I/O is performed in software by a qemu process, and on the mailing lists they recommend you pin that to a processor to improve performance... but that's a poor hack, and doesn't really work.

VMWare server may not scale but it's ten times better than Xen for this. ESX or ESXi beat it sideways.

Not in my experience, but, anyway, I don't use Xen on a full virtualized setup. I simply can't see the point if I can use it paravirtualized. As Windows 2008 can be run in a paravirtualized setup, I believe that you won't have any IO performance issues, since the guest OS will not have to handle any qemu process. Maybe. As I said, I never used Xen with a full virtualized setup outside the lab. This guy reached the same conclusions. "When the host OS can be modified, much higher performance numbers are obtained"[...]

Re:excellent sales story

[Anonymous Coward]

Direct all FreeBSD as a desktop users to PcBSD. It'll preserve your sanity.

Re:excellent sales story

[SanityInAnarchy]

According to TFA, which finally seems to be up, they actually considered this, and ran the entire thing on an NFS mount from a non-virtualized host.

What's particularly odd is that they don't actually mention trying a local, virtualized disk first -- they just rejected it out of hand. I would imagine NFS carries its own overhead...

BSD jails are actually sounding kind of cool, though.

Re:excellent sales story

[raynet]

This benchmark shows that Xen + Linux paravirtualization is faster than ESXi + Linux and as fast as ESXi + BSD: http://ivoras.sharanet.org/blog/tree/2009-04-25.some-virtualization-benchmarks.html

And I recall some other benchmarks with heavy I/O where paravirtualised Linux blew VMWare and other "real" virtualizations away.

Re:excellent sales story

[NeoSkandranon]

I think this is a really nice summary of most of the sales profession.

Re:excellent sales story

[awpoopy]

I was not aware of that. Thank you for the post.
I'll give it a try. It is too bad that it requires oracle, however there's some hope a little light at the end of the tunnel at least.

Re:excellent sales story

[rgviza]

The problem with x86 is I/O capability. Virtualizing servers on an x86 box compounds this weakness. Running a database on an x86 VM is sheer stupidity since a database server is an I/O bound process. It's the worst possible way to run a database server. In my shop we use VM's but only where appropriate. Production database server is not such a job.

I don't need to even try it to know this. If someone here suggested we do something like this for a production environment, and was serious, it would definitely throw a bright red flag to my manager and I.

-Viz

Re:excellent sales story

[DuckDodgers]

You're making an extremely solid argument for good virtualization, backups, and failover with redeployment.

You're making a far weaker argument for a proprietary solution that does those things. If some or all of your environments require Windows, then obviously you need a proprietary solution. But properly configured, backups, VMs, and fast switching between different setups in BSD jails, Xen, KVM (Linux Kernel-based Virtual Machine, not Keyboard-Video-Mouse switching), and other options give you the same flexibility. And a competent developer/admin can set them up for less than $15-$20k.

Re:excellent sales story

[Brett Diamond]

If your requirements are such that your "virtualized" machines can all run FreeBSD, then there is no reason not to leverage the jail system. It is robust, fast, and very secure. The current version of FreeBSD has added features to jail making it even more attractive (as well as simple to use and maintain). And since you are running FreeBSD, you have access to CARP, providing load-balancing and failover, as well as ZFS, the bee's knees of file systems.

Compare this with a vm solution (pick your favorite), where not only do you have to maintain multiple OS instances (one per VM) but also have to maintain the vm structure itself. I am not claiming that either of these tasks are that onerous, rather that they are tasks that are not required in the jail solution. And what do you get in return for these extra steps that you do not get with a jail? By all means, if there is something that a vm solution provides that cannot be done in a jail, then go for it; but dismissing jails out-of-hand doesn't serve anybody.

If all of your virtual machines can fun FreeBSD, the question really becomes, "why introduce levels of complexity when they are not required, impact performance, add security risks, and make the overall system more difficult to maintain?" The problem with a FreeBSD jail/carp/zfs solution isn't that it is inferior in any way; rather that, looking at OS popularity, FreeBSD is a distant third (forth really, but virtualizing Mac OS breaks licenses (and I don't really know where Solaris fits)) so FreeBSD is frequently not an option as it may not meet the requirements of the application. This may be mitigated with the Linux compatibility package and/or Wine; but you may want to run your Linux applications on a Linux machine (virual or otherwise), and likewise with your Windows apps.

Re:excellent sales story

[Michael Hunt]

You are woefully uninformed.

If you are using the (closed source, free) Citrix XenServer, the "Xenserver Guest Additions" ISO will install paravirtualised disk and network drivers, which handle IOs through a hypercall interface instead of qemu-dm's emulated devices (which, I'll grant, is hideously slow.)

If you're using GPL Xen, then the drivers at http://wiki.xensource.com/xenwiki/XenWindowsGplPv (that page points at source code, there are MSIs floating around, ask your distributor) accomplish the same ends. I've used them in the handful of cases I've needed to virtualise Windows at home. They're on par throughput-wise with the Citrix stuff I use at work.

Either way, such an approach gives IO performance on par with ESX, and the ability to mix and match this sort of VM with paravirtual Linux and Opensolaris guests on the same cluster of hosts buys you a shit ton in flexibility (IO performance in paravirtual guests exceeds anything that fully virtualised Xen or ESX will ever achieve, period, end of story.)

Re:excellent sales story

[wastedlife]

One thing we are using VMWare Server for is to get rid of some old, failing Win2k server boxes that we needed for a specific bit of software that wont run on 2003 or higher. The software is too expensive to upgrade for the amount of use it gets, but we cannot get rid of it entirely. So, we took the new 2003 R2 servers and threw VMWare Server on it to run Server 2000 and that app. We may eventually move to ESXi or XENServer, but for now this works just fine and did not disrupt the current functions of the 2003 server.

Re:excellent sales story

[Ant P.]

like this!
+----------+
|  PLEASE  |
|  DO NOT  |
| FEED THE |
|  TROLLS  |
+----------+
    |  |
    |  |
  .\|.||/..

Re:excellent sales story

[jra]

"The X Window System".

"XWindows" is explicitly deprecated in all the documentation, presumably to make Microsoft happy, though I don't think anyone ever admitted to it in public.

Re:excellent sales story

[OrangeTide]

Absolutely correct. The word VMware pushes around is "utilization".
Most people don't need a full machine to be a domain controller. And things like DNS servers, at least for most of us, don't need a full machine either.
I think I usually want a full machine if I'm doing a database though.

Re:excellent sales story

[OrangeTide]

They weren't looking for performance, but the performance was so terrible they had to mention and compare it. Because it was becoming a problem.

I suspect they were hoping that a VM running two databases would performance half as well as one database. Which is the common sense approach, but the reality is that the performance is so very much worse.

Re:excellent sales story

[Anonymous Coward]

Linux is getting process checkpointing. And will likely get migration support shortly. So you can move your services that way too. Although a VM still my be more convenient.

Re:excellent sales story

[nxtw]

I disagree. I consider Xen to be a kernel which other kernels are modified to run inside of, it is just a guest kernel making requests(read system calls) to a hypervisor(a special sort of kernel) that then translates it into requests to the host kernel. But mostly I feel this way because of the way I/O is handled in Xen is very much unlike the way VMware does it (go find my resume, I used to be an ESX developer at VMware).

A Xen guest runs its own kernel, with its own scheduler, network and block device subsystems, and filesystems. A VMware guest runs its own kernel, with its own scheduler, its own network and block device subsystems, and filesystems.

So what if Xen and VMware provide different mechanisms for running guest OSes and providing I/O? The same end result is achieved. In either case, guest OSes are isolated from other guest OSes and communicate via block/SCSI and network interfaces, and the guest OS runs some sort of network, block, or SCSI drivers to communicate.

BSD Jails are just a more Unix way of virtualizing a set of processes than Xen is. Xen requires an entire kernel to encapsulate the virtualization, BSD jails do not. In my opinion that is where they differ the most, but that difference is almost unimportant.

Userland containers (jail, VZ, Zones) use one kernel with one scheduler, one set of syscalls, one set of drivers, etc. Different containers may see a different network devices and filesystems, and may only see their own processes, but they're all running under one kernel that is aware of everything and limits the processes belong to each container to that container's namespace.

Re:excellent sales story

[Omega996]

I use xen to host a number of light- to medium-load VMs, and I've been very happy with the performance. Windows server with the paravirt drivers perform very welll.

Re:excellent sales story

[Bourbonium]

This is something that is hammered at over and over in the comments at the end of the original article, as well as here on /. They apparently did no research into virtualization before launching into this ill-advised kludge. It took them so long to figure out that they were using the wrong technology, they could have saved themselves a ton of work just by doing some rudimentary investigation/evaluation of multiple virtualization methods before going down the VMWare Server road. There are better "free" options than the one they chose, and probably some more appropriate options than the BSD Jails solution they eventually used. Or they could have paid a consultant to advise them in the first place if they weren't such cheapskates. I'm a notorious skinflint myself, but I know that doing your homework in advance is a better use of resources than the trial-and-error fiasco they endured. And they did this in a production environment with their customers' live data! Something tells me this story will not drive new business to their door.

Re:excellent sales story

[niw]

Yes, VirtualBox supports direct access. You have to edit config files to do it but it does work, but the VirtualBox team does not recommend it. They instead recommend that you use iSCSI, which VirtualBox also supports to connect to SANs directly. You also have to edit config files or use the CLI version of the VirtualBox machine setup (VBoxManage addiscsidisk), the option is not available in the GUI.

Re:excellent sales story

[Omega996]

ugh, there sure are a lot of people throwing out 'use ESXi - it's free!'. ESXi only runs on certain hardware, so if you don't have that hardware, it's not even a valid choice. Real management of ESXi is not particularly wonderful without using VMWare's management software, and that's not free, as far as I know.
I agree that performance-wise, it's a better choice than VMWServer. But I don't think the entry point for ESXi is as low as xen or Citrix's XenServer. XS runs on a wider range of hardware than ESXi, and the 'basic' management tools are pretty good, and also free.

I agree with everything else you mention, though, so don't think I'm trolling your post. I read TFA, and wow... where to start?

Re:excellent sales story

[Omega996]

that's BS - I use xen to host a mix of debian and Windows images, and the I/O performance is pretty good. VMWS is the 'option' if you want to have absolutely substandard I/O performance.

Re:excellent sales story

[Anonymous Coward]

Are they really "bare metal hypervisors"? There's no redhat hidden beneath the UI? No Linux Kernel for all the drivers?

Re:excellent sales story

[Omega996]

I have a T61 (6465-57U) and run VirtualBox on top of Ubuntu 9.04 to run instances of XP - it works really well for me, provided I stay away from 'seamless' mode. I only wish the T61 had a higher RAM cap. Performance-wise, though, it works pretty nicely.

Re:excellent sales story

[Znork]

Actually, drbd is active-active these days, so it would suffice on a HA basis.

LH sounds pretty neat, just a bit on the pricey side. Then again it's not like NetApp and the various SAN vendors offer particularly competitive pricing either.

Re:excellent sales story

[ArsonSmith]

If you have physical machines connected to the same SAN, both VMWare's products and Microsoft's Hyper-V support running failover clustering. This way, if one of the machines goes down, the VM and its services keep running with perhaps a small delay (in milliseconds) while the handoff to the other machine takes place.

VMWare has only introduced this capability in vSphear 4.0 (ESX 4.0) called Lockstep which runs a persistent vmotion between two different hosts so if one goes down the other takes over instantly. Microsoft's Hyper-V has nothing like this yet. What you are thinking of, regular vMotion, does not allow the ESX host underneath to die and keep the server running. You still have to boot and recover the VM on another ESX host in the cluster just as if the VM had crashed. Although this is much quicker and can be done before the failed ESX host is fixed.

Re:excellent sales story

[OrangeTide]

You didn't read the post. You missed:

Xen requires an entire kernel to encapsulate the virtualization, BSD jails do not.

Just repeating your same "point" over and over again while ignoring the post doesn't make me want to agree with you any more.

Re:excellent sales story

[atamido]

Xen says you simply need a CPU that supports VT (or whatever the AMD equivalent is) to run Windows VMs. You should grab a copy of Vista/2008/7 to install and test as you don't need a license key or activation for a few weeks. We haven't had any issues running them ourselves.

Re:excellent sales story

[Thundersnatch]

I thought DRBD was only active/active if you ran different services (volumes, NFS shares, whatever) on each of the two active nodes. I don't think you can share the same iSCSI volume from both nodes in a DRBD cluster, can you? That's pretty inefficent and hackish, becasue you have to decide which node is primary for a particular volume; reconfiguration would be manual and likely require downtime.

In any case, the LeftHand gig is N-way plus M redundancy. So you get to use the aggregate network bandwidth and disk IOPS of all of the cluster nodes at the same time for each volume. iSCSI initiators that support miltple connections per session don't even have to deal with sending everything to a "virtual IP"; they can send direct to the node that contains the interesting blocks. Note that VMware doesn't do that bit just yet, but they have iSCSI plug-ins for Windows (and I think Linux) that do. iSCSI HBAs can also do that I believe.

One high-end SAN feature LeftHand doesn't have is IO proritization per volume... they expect you to handle that from the initiator side, which doesn't work when the initiators are different machines. And support has gone downhill since the HP takeover - I need like 3 separate HP website acocunts just to download patches.

Re:excellent sales story

[geniusj]

I'm going to assume you work for a mid-large business where most of the servers are idling most of the time or where most processing is batch and not real-time. Virtualization can be a benefit in these types of environments, but it doesn't fit everywhere. I'd hope that you wouldn't go to google, for example, and suggest that they move their servers to virtualized infrastructure.

Google is actually a great counter-example to your arguments here. If you have the right processes in place and put some actual thought into your infrastructure, the result can be very manageable. But I understand that a lot of companies don't want to pay for the talent to put these kinds of things in place and find it much cheaper to buy a software package from VMware. They'll just pay a performance cost, but for many of them, it doesn't really matter.

It kind of reminds me of the JVM/CLR vs native code arguments.

Re:excellent sales story

[dvs01]

Your comment was a good read. You're someone that has sense.

Re:excellent sales story

[mysidia]

Hardware from around the time Win2k was a recent OS is so outdated by now, that it's not surprising VMware Server would be fine.

I would guess the functions of the servers are also small scale... An instance running on VMware Server using conventional (today's) hardware is indeed very superior to running on native 10-year-old hardware..

Re:excellent sales story

[mysidia]

By that logic, VMware server isn't free, becasue you have to pay for a copy of Windows to run it on top of.

ESXi pretty much runs on the hardware provided by all the major server manufacturers: Dell, HP, IBM, Sun. In fact, it can run on older hardware that Xen won't run on (because the processor lacks VT).

It's true you can find hardware it won't run on: most of this is whitebox hardware. Servers that utilize a RAID controller ESXi has no driver for.

(You can get around lack of DAS controller support by using a NAS device, e.g. a dedicated old server running OpenSolaris to expose your storage over iSCSI.)

Yes, if you tried to cut corners with your servers and buy off-brand generic hardware not supported by a major vendor to try to save some $$$. You may wind up having to pay that back, either get right hardware, buy ESX instead, or use Hyper-V.

I'd still pick Hyper-V server over VMware Server in a heartbeat, and pretty much, any server that works with Windows 2008 should work with Hyper-V server.

Re:excellent sales story

[mysidia]

They're building a vCenter for Linux, which is currently in Beta. It is only a matter of time...

Re:excellent sales story

[Omega996]

I don't agree re: VMware Server - I suspect it's much easier to find a computer with a copy of Windows XP on it than it is to find a server with the specific hardware requirements for ESXi. Performance, though... *shudders*
As far as white-box hardware, you're right. In my particular case, however, the machines in question were HP servers, not generic corner-cutting hardware, as you imply.
Further, on the two ML-series servers I had that met the hardware prerequisites for ESXi, neither would boot ESXi without a panic. I installed xen and had no problems.
I never tried Hyper-V, as xen met our performance expectations. How is Hyper-V's support for non-MS operating systems?

Re:excellent sales story

[ckaminski]

It was a J2EE development environment, where the app, on startup, consumed 1.5 GB of memory, before doing any calculations. Some also had oracle servers running on them. We had 16-24GB of memory on each VM host, and overcommitted by about 100GB.

Each VM was the same app, as each developer was testing their fixes. Because we had a lot of ESX servers, we mixed some production VMs on these too (at higher resource priority), to spread the load around a bit. We didn't have the budget for separate production and development ESX clusters.

Re:excellent sales story

[thanasakis]

You mean Solaris Zones. LDoms is a different thing.

Re:excellent sales story

[wastedlife]

This is true, it is one app that is used sparingly by special ed, but cannot be currently upgraded or removed. VMWare server provided exactly what we needed to get rid of the 7+ year old servers. Also, I've done some testing with XenServer, and the memory overhead of the hypervisor for that seems pretty high. Granted, the VMs themselves run at near-native performance, but it seems to be a pretty significant RAM cost. With only 2 virtual machines, the RAM usage for XenServer itself was hovering between 500 and 600 MB. VMWare server plus the Win2k virtual machine is using less than that right now inside the 2003 R2 server.

One thing I will note is that running several more virtual machines does not seem to significantly increase memory usage by XenServer, so it might be just a matter of scale.

Re:excellent sales story

[mysidia]

I've run ESXi on various HP ML G5 servers without issues...

Yes, there are some servers ESXi would panic on, purple screen. I think much of that may be fixed in ESXI update 4, because I see less of that.

And in general, the newer releases seem to support more hardware.

In general, my experience was, that on most servers that had problems like that, I upgraded the BIOS ROM flash, BMC flash rom, and RAID controller flash ROM, to a version that was a little recent, and suddenly ESXi started working.

Either that, or I used a well-publicized hack (first boot the system with acpi=off, after installation boot the system with an Ubuntu rescue CD), and manually edit .cfg files to disable ACPI.

But certainly you're best advised to pick hardware from the VMware HCL.

If you plan your servers for virtualization from the start, this should not be a problem, you get servers that your hypervisor works best with, preferably you choose models it's certified on.

If you have existing servers you want to use, instead of picking hardware that works well with your hypervisor(s) of choice, then you didn't design your latest upgrade cycle to be compatible with a full-scale conversion to virtualization.

Re:excellent sales story

[mysidia]

XenServer and Hyper-V both have very limited support for non-MS OSes. It's the major reason I prefer VMware.

XenServer seems to be good, only if Linux and Windows are what you want. Hyper-V is much the same way.

The added caveat is SuSE Linux is the only Linux distribution officially supported by Hyper-V. Other virt platforms officially support Redhat, and VMware even provides VMware Tools for some lesser-used non-Enterprise distros like Fedora and Ubuntu.

However, you can definitely make Redhat work in Hyper-V. And there are directions online for integrating the Hyper-V tools into your Redhat system. The downside is you probably have to compile a kernel to do it.

There are many BSD-based virtual appliances that work just great in VMware, but do not work in Xenserver.

I have tried various versions of Solaris, none of them install properly in XenServer. They all work in VMware without any extra effort.

I cannot figure out a way of getting FreeBSD, NetBSD, or OpenBSD to run correctly in a XenServer virtual machine, either it won't install correctly in HVM mode, or there are various kernel driver issues that make it not work correctly; e.g. virtual network card randomly bounces, or the total sustained I/O throughput that can be achieved is outright horrible).

In HVM mode:
FreeBSD - XenServer: installer doesn't work, the installer doesn't even boot, seems to enter some sort of infinite loop at the bootloader.
NetBSD - Installer loads, gives very encouraging signs, just when installation is about to start, the kernel panics in a 'gzip' process and drops to a DB prompt.
OpenBSD - Installer works just fine, system seems to work (most success i've had with any *BSD), until you start generating network traffic, then the interface starts resetting every few seconds, it loses pings and all TCP connections freeze every few secs basically, throughput if you FTP or perform file transfers is horrid.

It's true there are domU kernels available for some of the BSDs, but XenServer doesn't actually let you upload a kernel and point a config file to it. I tried for days, could not make that work.

Re:excellent sales story

[Znork]

Sorry for taking such time to reply, I was away for a bit.

I thought DRBD was only active/active if you ran different services

Nope, you can configure DRBD volumes to be active/active on both nodes at the same time, from version 8. From version 9 it looks like they're implementing more than 2 mirrors too. The usual caveats of shared disk semantics apply, of course; you need GFS or OCFS or something on top, be sure you're not writing in two places at once, or something else to ensure consistency.

So iSCSI sharing from two simultaneously active nodes sharing the same mirrored image is certainly possible, you'd just have to switch to the other iSCSI target. How does LeftHand manage consistency? For example, if power gets lost on a node and writes get redirected, how does it ensure there were no in-transit io's going through the dead node? Or does it take the latency hit and not return IO complete until the write is committed on the other nodes?

Still, while it's certainly possible to do, for simplicitys sake I'd personally probably just share separate volumes and mirror on the mounting hosts.

iSCSI initiators that support miltple connections per session don't even have to deal with sending everything to a "virtual IP"; they can send direct to the node that contains the interesting blocks.

Hmm, that's an interesting feature. I'll have to look over if the iSCSI initiators I'm using can do multiple connections :).

I need like 3 separate HP website acocunts just to download patches.

Agh, yah, been there done that.

Re:excellent sales story

[Thundersnatch]

So iSCSI sharing from two simultaneously active nodes sharing the same mirrored image is certainly possible, you'd just have to switch to the other iSCSI target. How does LeftHand manage consistency? For example, if power gets lost on a node and writes get redirected, how does it ensure there were no in-transit io's going through the dead node? Or does it take the latency hit and not return IO complete until the write is committed on the other nodes?

Well, that is at least part of their special sauce I suppose. I assume they take the latency hit to a degree. Our testing showed spindle-level IOPS throughput overall at longer queue depths, but higher latency than other more traditional systems we demoed running IOmeter. We also yanked cords, power, and jiggled a known-bad network cable during heavy write tests and never observed missing or corrupt data. Some of that may have been covered up by retries on the client side though (haven't read the iSCSI specs to see what is requried of iSCSI clients in that area).

Interesting

[kspn78]

I wonder if this would help me, I am running 2 VMWare servers on an older box and it is a little lethargic at the moment. If I could ever get to the story I might be able to find out :|

Re:Interesting

[symbolset]

If by "Older box" you mean more than 3 months old, it's time to upgrade :)

Re:Interesting

[machine321]

Don't use an older box, get a newer box with a CPU that does virtualization. That makes all the difference.

Re:Interesting

[ForAllTheFish]

Just try ESXi if your hardware will support it.... we have it running at work like a dream.

Re:Interesting

[Kz]

Don't use an older box, get a newer box with a CPU that does virtualization. That makes all the difference.

1: no CPU 'does virtualization', unless you're talking about bit IBM's POWER machines, or such
2: VMWare Server doesn't use current processors' VMX / SVM extensions, so they don't make _any_ difference.
3: Xen and KVM do use them, in fact can't run windows (or at all in KVM's case) without them. so the difference isn't about performance, is about being able to do it.

Back to the Future?

[guruevi]

So we go back to where we started from: chroot and jails. What really is the benefit of extended virtualization? I haven't "embraced" it as I am supposed to do.

I can see where it makes sense if you want to merge several servers that do absolutely nothing all day into a single machine but a decent migration plan will run all those services on a single 'non-virtual' server. Especially when those machines are getting loaded, the benefits of virtualization quickly break down and you'll have to pay for more capacity anyway.

As far as high availability goes: again, low cost HA doesn't work that well. I guess it's beneficial to management types that count the costs of but don't see the benefit in leaving a few idle machines running.

Then you have virtualized your whole rack of servers into a quarter rack single blade solution and a SAN that costs about the same than just a rack of single servers but you can't fill the rack because the density is too high. And like something that recently happened at my place: the redundant SAN system stops communicating with the blades because of a driver issue and the whole thing comes crashing down.

Re:Back to the Future?

[hpavc]

Comparing ESX and Zones? Seems like a horridly thought out comparison

Re:Back to the Future?

[Anonymous Coward]

Virtualization, as far as it is applied to server software, is a kludge. It is a way of making software work on one machine which would otherwise have conflicting OS and security requirements. The virtualization layer provides the abstraction and isolation which should be provided by the OS but isn't. The reason is API complexity: Virtualization deals with a relatively small and low level API. The operating system has a much broader API with much more complex dependencies, so it is much harder to secure and test for incompatibilities.

It should not surprise anyone that removing redundancies (to save money) likely also decreases fault tolerance. On the other hand, it is often beneficial to remove "accidental" redundancy and add redundancy back in with a plan.

Re:Back to the Future?

[ckaminski]

Consolidate several lightly used, different services onto ONE server? Have you ever managed multiple applicatoins in a heterogenous environment? Consolidating applications causes operational complexity that is inappropriate in a lot of instances. While service isolation is easy on Unix platforms, it's not on Windows.

Re:Back to the Future?

[wrench turner]

Running multiple services on one OS requires that when you must reboot a server because of an OS bug or mis-configuration all of the services are brought down... Same if it crashes or hangs. As compelling as that is I've never used a hypervisor in 30 years on 10's of thousands of servers.

I do routinely use chroot jails on thousands of servers to isolate the application from the host OS. This way I do not need to re-qualify any tools when we implement an OS patch.

Check it out: http://sourceforge.net/projects/vesta/ :-)

Re:Back to the Future?

[Mr. Flibble]

So we go back to where we started from: chroot and jails. What really is the benefit of extended virtualization? I haven't "embraced" it as I am supposed to do.

I can see where it makes sense if you want to merge several servers that do absolutely nothing all day into a single machine but a decent migration plan will run all those services on a single 'non-virtual' server. Especially when those machines are getting loaded, the benefits of virtualization quickly break down and you'll have to pay for more capacity anyway.

This is exactly what VMware lists as best practice for using virtualization. If a server is maxing out, it should not be virtualized as it is not a good candidate. However, if you have a number of servers that are under utilized, then the advantage of turning them into VMs become clear. VMware has a neat feature called Transparent Page Sharing, where VMs using the same sections of memory with the same bitmaps across the same images are all condensed down into the same single pages of memory in the ESX server. This means that your 10 (or more) windows 2003 server images "share" the same section of RAM, this frees up the "duplicate" RAM across those images. I have seen 20% of RAM saved by this, IIRC it can go above 40%.

As far as high availability goes: again, low cost HA doesn't work that well. I guess it's beneficial to management types that count the costs of but don't see the benefit in leaving a few idle machines running.

If you mean VMware HA, I find it works quite well, granted the new version in Vsphere (aka Virtual Center 4) is much better as it supports full redundancy.

Then you have virtualized your whole rack of servers into a quarter rack single blade solution and a SAN that costs about the same than just a rack of single servers but you can't fill the rack because the density is too high. And like something that recently happened at my place: the redundant SAN system stops communicating with the blades because of a driver issue and the whole thing comes crashing down.

You are assuming that the people don't have this already. I have been to a number of data centers that have racks and racks of under-utilized machines that also have SAN storage. VMware Consolidation is a way of consolidating the hardware you already have to run your ESX hosts. You use a program called VMware Conveter to do P2V (Physical to Virtual) to convert the real hardware machines to VMs, then you reclaim that hardware and install ESX on it, freeing up more resources. You don't always have to run out and buy new hardware!

VMs are great when the hardware is under-utilized, I do not recommend VMs that max out, and neither does VMware.

Re:Back to the Future?

[gdtau]

"What really is the benefit of extended virtualization?

1) The ability to deploy a system image without deploying physical hardware. All those platforms you are meant to have, but don't: a build machine, an acceptance test machine, a pre-production test machine. And if you've done all the development and testing on a VM then changing the machine when it moves from production from a VM to being real hardware doesn't seem worth the risk.

2) IT as a territorial dispute. You are the IT Director for a large enterprise. You want everything in good facilities, what after the last time a cleaner unplugged the server that generates customer quotes, bringing revenue to a screaming halt. The owner of the quotes server will barely come at that. They certainly won't hand over sysadmin control. Their sysadmins like whitebox machines (the sysadmin's brother assembles them), but you'll never have parts on the shelf for that if it breaks. So get them to hand over a VM image, which you run on hardware of your choice, and which you can backup and restore for them.

3) Single hardware image. No more getting a "revised" model server and finding that the driver your OS needs isn't available yet (or better still, won't ever be available for that OS, since the manufacturer really only supports new hardware in their forthcoming releases). And yeah, the server manufacturer has none of the previous model in stock.

And of course there's minor stuff. Like being able to pull up a shiny clean enterprise image to replicate faults.

You'll notice the lack of the word "silver bullet" above. Because virtualisation isn't. But it does have a useful role, so the naysayers aren't right either.

I'm waiting for the realisation that merely combining images onto one physical machine does not do much to lower costs. For a directly-administered Windows OS the sysadmin's time was costing you more than the hardware. Now that the hardware is gone can you really justify maybe $50kpa/5 = $10pa per image for sysadmin overhead? This is particularly a problem for point (2) above, as they are exactly the people likely to resist the rigorous automation needed to get sysamdin per image overhead to an acceptable point (the best practice point is about $100 per image -- the marginal cost of centrally-administered Linux servers. You'll notice that's some hundreds of times less than worst-practice sysadmin overhead).

I'll also be a bit controversial and note that many sysadmins aren't doing themselves any favours here. How often do you read on Slashdot of time-consuming activities just to get a 5% improvement. If that 5% less runtime costs you 5% more sysadmin time then you've already increased costs by a factor of ten.

Re:Back to the Future?

[sumdumass]

I think he was looking for the VMware argument over jails and so on. Consolidating lightly used services probably means getting rid of 5 dual core zeon power edge server and running virtual machines on one while duplicating them on another for availability. I don't think he meant loading five different versions of accounting applications on the same server OS.

Re:Back to the Future?

[BitZtream]

This means that your 10 (or more) windows 2003 server images "share" the same section of RAM, this frees up the "duplicate" RAM across those images. I have seen 20% of RAM saved by this, IIRC it can go above 40%.

Theory is so nice isn't it, in practice those 10 servers will be sharing about 2 megs of pages, really useful isn't it!

You don't always have to run out and buy new hardware!

Yea, you just spent twice as much paying for VMware licenses, YAY totally worth it.

Re:Back to the Future?

[Lord Kano]

I can see where it makes sense if you want to merge several servers that do absolutely nothing all day into a single machine but a decent migration plan will run all those services on a single 'non-virtual' server. Especially when those machines are getting loaded, the benefits of virtualization quickly break down and you'll have to pay for more capacity anyway.

When you let "managers" make decisions about other departments you run into problems with this. "These servers is sitting at near idle 99.9% of the time we should consolodate them." But if these servers are primarily used at week-end, month-end or year end closings. All of a sudden you've got one server thrashing instead of four servers smoothly processing all of their transactions.

LK

Re:Back to the Future?

[mlts]

One other benefit of extended virtualization is that the host OS can offer some security measures that guests can't.

For example, if I am running Windows Server 2008 on hardware that supports it (TPM + VM extensions in the CPU), I can use BitLocker or TrueCrypt to encrypt the system disk and the disks that the VMs run on.

This way, if I'm using an app that is in a MS-DOS VM on a laptop, if the laptop is stolen, the data inside is protected, even though MS-DOS usually doesn't have that much in the start of the art disk encryption (perhaps an old version of Stacker that supports password protection, but that's basically it.)

Re:Back to the Future?

[Mr. Flibble]

Theory is so nice isn't it, in practice those 10 servers will be sharing about 2 megs of pages, really useful isn't it!

Given that the ESXi server that I personally have as my own test bed, which has dual 3.0 Ghz Xeons and 8 GB of RAM, yes that is a hell of a lot.

The VMware lic's are not NEARLY as much as the cost of new hardware. The ESXi box I have is 4 years old.
And, FYI, it is the licensed version of ESXi installable 3.5, not the free version.

Don't forget, depending on the type of windows licenses you have, if it is per-processor based, this means I can run all 10 of my VMs on only 2 lic's from Microsoft. (Because each VM only uses 1 of the 2 cores). Getting 8 "free" Windows 2003 server lic's is a pretty damn good deal.

Don't disparage Vmware just because the sell a product for money. In most cases where I have seen it used it has resulted in a savings overall. Yes, you can screw it up and cause it to cost you more, but that goes for pretty much everything.

Re:Back to the Future?

[bensch128]

While service isolation is easy on Unix platforms, it's not on Windows.

I don't think it's even that easy on unix. I had to install a lot of custom packages on our Suse machines at work to get reviewboard up and running.
With that level of customization, I won't dare try to get another service running on that VM.
Also, another awesome benefit to VMWare's concept is that it's trivial for me to clone the machine and setup an identical machine to test doing an upgrade.
After that, I'm a fan of VMWare server. Of course, the service is not bandwidth intensive so the issues are different.

Re:Back to the Future?

[miro f]

"What really is the benefit of extended virtualization?

1) The ability to deploy a system image without deploying physical hardware. All those platforms you are meant to have, but don't: a build machine, an acceptance test machine, a pre-production test machine. And if you've done all the development and testing on a VM then changing the machine when it moves from production from a VM to being real hardware doesn't seem worth the risk.

This is the scary bit here. Not so much that you might think the risk of moving from virtual to bare metal might not be worth it, but that often the decision is made that the risk (or effort) of moving it out of the test environment isn't worth it.

Then imagine what happens when you shut down a server in a test lab (or worse, a machine that's sitting in the corner of an office somewhere) to try to contain a virus infection and you get a phone call from someone 5 minutes later saying they can't do any work because their system went down.

And if you're thinking "happens all the time in small business" I'm talking over 30,000 employees worldwide.

Re:Back to the Future?

[bertok]

Don't blame the technology on the idiots that implement it.

If SAN cost is an issue, explain to management that 'reassuringly expensive' is not the only metric by which SAN hardware should be evaluated. ESX has native NFS and iSCSI support if you're really cheap.

SAN failures will bring down physical hardware too, as will internal drive failures. A network switch failure - ditto. Sounds like you had a single point-of-failure where you shouldn't have had one, which was a design error. The failure of a single 'driver' shouldn't be able to take out an entire chassis, ever. The other stupid thing that I've seen is people making one enormous LUN for the ESX cluster. That's like one enormous basket with a metric ton of eggs in it.

By the sound of it, you've forgotten to add in the hidden costs of managing hardware. Everything from "we don't want to migrate software 'x' off the Pentium III because we're afraid to touch it", to "oops, the server went pop, lets fuck around with tapes for two days while the business grinds to a halt". How long does it take you to build a server, from scratch, all the way up to production functionality? A day? Two? How much does that cost your company?

What the 'management types' see are the very real advantages of virtualization, especially with ESX, which allows some amazing things.

ESX allows you to upgrade the host hardware, replace the SAN, the SAN switches, the network switches, in fact, EVERY SINGLE DEVICE in your data center while every virtual machine keeps running, uninterrupted.

I've seen a server with 470 days uptime running on a cluster with no hardware component older than 3 months. That's so right, it's almost wrong! 8)

ESX doesn't stop there, you can do an upgrade of a shared-disk cluster live, with virtual machines powered on and running. You can go from, say, v3.0 to v3.5 without stopping anything. I've seen cluster upgrades where you could upgrade the cluster-shared filesystem itself (VMFS) while it was in use.

That's just the start of it. You can build a cluster out of mixed hardware, running ancient virtualized NT4 boxes with horrendous custom applications and cluster everything with a "drag & drop" operation. Not just 'cluster' in the stupid Microsoft sense of "we reboot from a crash slightly faster", but real, shared-disk clustering, where VMs can balance between hosts on-the-fly. I've seen a screenshot showing a history of 160,000+ VM host-to-host migration operations on a production system! You can now get instant, zero-loss failover of a VM by ticking a checkbox. It'll automatically mirror the memory and replay all inputs on both ends in-sync, so if you lose a host, failover is instant, network connections are not interrupted, and no transactions are lost.

So... how do you manage your data center?

Re:Back to the Future?

[asdf7890]

Don't forget, depending on the type of windows licenses you have, if it is per-processor based, this means I can run all 10 of my VMs on only 2 lic's from Microsoft. (Because each VM only uses 1 of the 2 cores). Getting 8 "free" Windows 2003 server lic's is a pretty damn good deal.

Erm, I'm pretty sure it doesn't work like that - I recommend that you go find and analyze the small-print to make sure you are covered in case someone comes round to audit!

My understanding is that each virtual CPU that Windows runs on would be considered a CPU for Windows licensing terms so if you have 2 1-to-2-CPU Win2K3 licenses then you are licensed to run Windows 2K3 in two VMs and no more (or use one license on the host and one in a VM). If you run 10 VMs each with Windows as the OS then you need 10 Windows licenses (if you buy each separately) or at least 10 CPU license (if you use some sort of bulk purchase arrangement for per-CPU lics).

Also, the "1 or 2 CPU" term in a lot of MS licenses only covers one or two CPUs in the same machine, not running with the same license on two separate single CPU machines (physical or virtual). They don't count cores (just physical CPU packages) so you would be OK with a "1-2 CPU" license on a machine with two quad-core CPUs, but I don't know how this extends to VMs (they are likely to see 4 vCPUs in a VM as 4 CPUs not 4 cores on one CPU, irrespective of what arrangement of physical CPUs/cores the host machine has).

It is a while since I reviewed the licensing terms for Retail/OEM Windows Server releases (at work we are a small MS dev shop, but our Windows servers and desktops came with there own lics where needed (or run Linux in the case of file servers and VMWare host machines) and the OS installations and those we use (on physical boxes or VMs) for testing are "licensed" via our MSDN subs), so I could be wrong here. But I don't think I am...

Re:Back to the Future?

[dkf]

I'm waiting for the realisation that merely combining images onto one physical machine does not do much to lower costs. For a directly-administered Windows OS the sysadmin's time was costing you more than the hardware. Now that the hardware is gone can you really justify maybe $50kpa/5 = $10pa per image for sysadmin overhead? This is particularly a problem for point (2) above, as they are exactly the people likely to resist the rigorous automation needed to get sysamdin per image overhead to an acceptable point (the best practice point is about $100 per image -- the marginal cost of centrally-administered Linux servers. You'll notice that's some hundreds of times less than worst-practice sysadmin overhead).

I'll also be a bit controversial and note that many sysadmins aren't doing themselves any favours here. How often do you read on Slashdot of time-consuming activities just to get a 5% improvement. If that 5% less runtime costs you 5% more sysadmin time then you've already increased costs by a factor of ten.

You shouldn't be hiring worst-practice sysadmins. Utter false economy.

The big savings with virtualization are in improving utilization of machine rooms; it's much better to have one physical machine hosting 5 VMs than 5 separate machines. The savings are in terms of floor-space, electricity and cooling, all of which can really stack up over time.

Re:Back to the Future?

[herring0]

The Windows 2003 Enterprise license allows up to four installed Windows 2003 Server (up to enterprise, but can use standard as well) virtual machines. However, as I understood it that only applies if you are using the Microsoft virtualization technologies. The results of an audit may vary when you are using the VMWare for your physical box.

Unfortunately in terms of licensing many of the companies are only just catching up and depending on your enterprise or auditors they may or may not be 100% up to date on these issues. Hard to blame them though since some of the vendors are changing how they count processors and machines what seems like daily.

Re:Back to the Future?

[gollito]

That may have been true with old licensing but if you purchase any new licenses they all come with "virtual machine" licenses of some sort.

Windows Enterprise allows you to install that copy of Windows four times on the same physical hardware. If you buy Datacenter (which is licensed per socket on the physical machine) you can install as many copies as that physical hardware can handle.

And yes this licensing applies to any hypervisor not just Microsofts Hyper-V. (link)

Re:Back to the Future?

[PitaBred]

Sysadmin time isn't the only cost of extra hardware. Don't forget the extra cooling and extra power draw, plus space in the data center that could be used otherwise. Consolidating even 5 lightly-used servers will save a significant amount of cooling and electricity energy. As long as the functionality doesn't change, you're saving a fair bit of money there even with the same number of admins.

Re:Back to the Future?

[Omega996]

you better go back and read the licensing terms for Windows Server 2003... Using the Enterprise Edition, you may run up to four 2003 servers with no additional license, provided you're using 2003R2 EE as the host OS. With Datacenter, you have unlimited licenses, provided that you're using 2003R2 DE as the host OS. Any other scenario, and you're required to license each copy of 2003R2.

Re:Back to the Future?

[Omega996]

The MS license 'graciously' allows you to use multi-core CPUs without extra cost (such nice bastards), provided the number of CPU sockets fits within your OS license. Regarding virtual procs and MS OSes - according to Microsoft: "For licensing purposes, a virtual processor contains the same number of cores and threads as its underlying physical hardware system."

Re:Back to the Future?

[Anonymous Coward]

If you really think virtualization is going to allow you to get away with less sysadmin time, you are insane. Virtualization often results in having more operating systems around than before, because you have the ability to have more now.

You were previously running 10 OS's on 10 underutilized systems. Now you can run 20-30 OS's on the same hardware, or you can get rid of half the hardware. Guess which situation is more likely?

Not only do you have to manage the virtual machines, but you also have to manage the host OS's now. So, you've gone from 10 OS's to 30-40. If you want HA, VMotion, and other niceties, you'll also have to have shared storage.

Don't forget that virtualization is a lot more complex than running OS's on bare metal. Virtualization doesn't allow you to get rid of sysadmins, it allows you to do more with the hardware you have.

(Good) Sysadmins aren't worried about this.

Re:Back to the Future?

[nurb432]

Sounds like you haven't got things setup properly. We have *hundreds* of high load VMs running on just a few ( super high end ) hosts all running off a SAN across 2 fiber channels to each host back to the SAN ( 2 for redundancy, not load balance ).

HA works perfectly as currently designed ( auto restarting of machines if a host drops out ). If you look into the new offerings from VMWare, it wont even drop a packet on a "protected guest" if a host goes down.

Ya i'm a fanboy, but they have a great product so its easy to be one.

Re:Back to the Future?

[Anonymous Coward]

> You don't always have to run out and buy new hardware!

Yes you have. Unless you want to run ESX on processors that don't have hardware support for virtualization. And if you try, you will be very, very sorry.

Re:Back to the Future?

[ckaminski]

I took it to mean that I'm putting Accounting + Tech Support + ClearCase + Leads Management + CRM + Email on one server. That's bad juju in ANY environment.

Re:Back to the Future?

[sumdumass]

Lol.. Yep, I agree that's bad. But running them in VMs on the same hardware but separate OS's seems rather acceptable.

Re:Back to the Future?

[ckaminski]

And the benefit comes when you add shared storage clustering. Now you have failover without having to implement native OS clustering (Though that can *STILL* be beneficial from a service upgradability standpoint).

UML FTW!

[morgan_greywolf]

Or there's always User-Mode Linux.

Re:UML FTW!

[Just Some Guy]

Yeah. Now imagine that virtualized processes run exactly as fast as "native" processes. Starting to sound pretty good?

Re:UML FTW!

[solafide]

UML is possibly the worst-maintained part of the Linux kernel. Don't try building it in any recent kernel. It won't compile.

Re:UML FTW!

[Antique Geekmeister]

And "mock" (http://fedoraproject.org/wiki/Projects/Mock), which I use to build software for one platform on another OS. The chroot cages it builds automatically are quite useful for getting clean, new compilation environments without the pain of allocating hardware, configuring device drivers, setting up networks, etc., etc.

Re:UML FTW!

[asdf7890]

Or there's always User-Mode Linux.

Good though UML is for some things (particularly its original reason for being created: certain driver/module development and testing work) I wouldn't bring it into a discussion about performance.

While I've seen reports that show UML can beat VMWare-on-a-host-OS arrangements (i.e. the Player and Server products) in some benchmarks where almost all the benchmark is in userspace (i.e. almost no system calls) the performance hit of a system call can quickly bog down most real world VMs. As one anecdotal data point: testing a Zimbra server in a UML VM resulted in much slowness including parts of the admin interface (not always fast at the best of times) being practically unusable, but the same installation in a VMWare VM (using the same set of test data (migrated from one VM to the other) to improve the fairness of the test) was significantly more responsive on the same host hardware+OS despite haivng less RAM allocated (so the VMWare guest OS ended up swapping a little where the UML one didn't). I saw similar results when moving a simple web server (originally a copy of a VM run on linode.com back when they ran on UML not Xen as they do now) from UML to VMWare.

UML has advantages in specific cases and was the best inexpensive option for a time if chroot wasn't right for your needs, but I wouldn't recommend it generally now considering the quality of current freely (and in some cases Freely) available alternatives.

-1, Flamebait

[Anonymous Coward]

TFA: "Error establishing a database connection"

So much for that. Also, am I correct in assuming BSD's jail is the equivalent of Linux's chroot? Is this another case of "Didn't know I should have been limiting processes instead of visualizing another OS for a single process" stories? I mean .. isn't that, well, obvious?

Re:-1, Flamebait

[eosp]

Well, the BSDs all have chroot as well. However, jails have their own sets of users (you can have root in one jail but not in the system at large) and the kernel makes more division between the data structures from jails (and the host system) than chroot does. In addition, ps(1) can only show in-jail processes, network configuration changes are impossible, and kernel modifications (modules and securelevel changes) are banned.

Re:-1, Flamebait

[larry bagina]

Linux's chroot is actually BSD's chroot. Bill Joy invented it.

Re:-1, Flamebait

[jelle]

Linux has that with: http://linux-vserver.org/

Both linux vservers and bsd jails have existed for plenty of years before vmware, xen, virtualbox, etc.

And, on that subject, this http://en.wikipedia.org/wiki/LinuxPMI is based on 'mosix', which made your cluster of linux boxes appear as one single massive machine, with transparent process migration and all that.

There are lots of virtualization and clustering options out there.

Re:-1, Flamebait

[Guy Harris]

Linux's chroot is actually BSD's chroot. Bill Joy invented it.

BSD's chroot is actually V7's chroot. Ken, Dennis, and company invented it.

Re:-1, Flamebait

[Anonymous Coward]

vserver first supported kernel 2.4 which was released in 2001...

vmware was founded in 1998...

Re:-1, Flamebait

[BitZtream]

chroot IS NOT A SECURITY WALL.

chroot is not a JAIL. How many times does some idiot have to be burned before people get the point.

FreeBSD jails are NOT anything like simple chroots.

Jails ARE security fences. chroot is just a way to make an app have its own unique directory tree with the libs and files it wants separate from the normal system, it does not prevent you from accessing files outside the chroot if you put a little effort into it. Its more or less a way to change what the root of the file system APPEARS TO BE when you use full paths. But it doesn't actually change the root of the file system or deny you access to it.

chroot 'changes' end at the file system, and its nothing more than what / references.

Please learn what chroot actually is before you ever mention it again like you have a clue, a good starting point is 'man chroot'

I really wish you people would get a clue before trying to act like an admin and talk about security features.

Re:-1, Flamebait

[eosp]

Um, perhaps it would be beneficial to read my post before screaming. I said basically what you said. In fact, the majority of my post was describing the security features that jails have and chroot does not.

Re:-1, Flamebait

[Anonymous Coward]

You really seem to like being right (or at least telling other people they're wrong).

I'd like to point out that ridiculing people and being obnoxious actually lowers the quality of discussion instead of improving it. If you still want to do that, please at least make sure you don't start the ranting in totally inappropriate places like you did now.

Re:-1, Flamebait

[Tony Hoyle]

Breaking out of a chroot on any modern system is damn near impossible without root access - and if someone gets root access you're hosed anyway.

It's a perfectly good solution for what it's intended for - limiting access to the filesystem. A chroot *is* a jail (in fact the term 'jail' was first coined in references to chroots).

Re:-1, Flamebait

[Omega996]

I realize this post is a bit hysterical, but the poster's correct in his/her information. why is this flagged flamebait?

XenServer worked for us

[gbr]

We had performance issues with VMWare Server as well, especially in the disk I/O area. Converting to XenServer from Citrix solved the issues for us. We have great speed, can virtualize other OS's, and management is significantly better.

Re:XenServer worked for us

[Anonymous Coward]

Shouldn't you be comparing XenServer to ESX instead of VMware's free hosted virtualization product? I don't see how the comparison here is fair. It's like saying Mercedes' Smart Car is too slow so you went to a BMW M3.

Re:XenServer worked for us

[00dave99]

XenServer has some good features, but you really can't compare VMware Server with XenServer. I have many customers that were impressed to be able to run 4 or 5 VMs on VMware Server. Once we got them moved to ESX on the same hardware they couldn't believe that they were running 20 to 25 VMs on the same hardware. That being said back end disk configuration is the most important design consideration on any virutalization product.

Re:XenServer worked for us

[ckaminski]

I broke VMware ESXs upper CPU limit of 168 vcpus with 104 running VMs. About 20 of which were under any significant load. 24ghz of CPUs and 32 GB of memory. Pretty damn impressive, if you ask me.

Re:XenServer worked for us

[machine321]

management is significantly better.

That usually solves a lot of performance problems.

Re:XenServer worked for us

[funwithBSD]

The company I work for has just about every Midrange VM solution you can imagine: Citrix, ESX (Seperate Windows and Linux clusters), Solaris Containers, and AIX VIO/Lpars. That is more or less the order of stability, btw.

of all the solutions, AIX is the most consistent and stable. Cheap is what they are not, but in our case they are Blue Dollars. It does exactly what it is billed to do, day in, day out.

Solaris 10 Zones a royal bastard to patch, but otherwise perfectly stable. (quite frankly, they are really just jails, just a little more configurable I suppose)

ESX is stable enough, depending on hardware. Certainly easier than anything but perhaps the HMC.

Citrix is the worst of the lot. But with so much invested, they don't want to do anything else.

Re:XenServer worked for us

[coffee_bouzu]

Comparing XenServer and VMware Server is like comparing apples and oranges. While VMware Server is impressive, it is very much like an emulator: It runs on top of another operating system and has to work harder to execute privileged commands. VMware ESX is a bare-metal hypervisor that is better optimized to do virtualization. While it is still doing "emulation", It is a much better comparison to XenServer than VMware Server is.

TFA is slashdotted at the moment, so I don't know if VMware Server or ESX is being compared. Either way, the advantage of virtualization is not performance, it is flexibility. The raw performance may be less, but it gives you the ability to do things that just aren't possible with a physical machine. The ability to hot migrate from one physical machine to another in the event of hardware failure or replacement and the ability to have entire "machines" dedicated to single purposes without needing an equal number of physical machines are, at best, more difficult if not impossible when not using virtualization.

Don't get me wrong, I'm no VMware fanboy. It certainly has its rough edges and is certainly not perfect. However, virtualization as a technology has undeniable benefits in certain situations. Absolute performance just isn't one of them right now.

Re:XenServer worked for us

[mosch]

Zones are like jails... if jails had something resembling resource management.

Re:XenServer worked for us

[BitZtream]

So you converted from Citrix to Citrix? Or was that XenServer to XenServer?

Re:XenServer worked for us

[funwithBSD]

Yeah, but what would you limit?

If your base machine is something modern like a T5220 or T5240 getting the CPU worked up is hard. especially if you are doing what it is designed to do: Java and Apache.

Memory is the limiting factor for us in almost all of the VM environments, not CPU.

Storage is never an issue, ZFS sees to that.

Re:XenServer worked for us

[gbr]

I agree, it is like apples and oranges, as is BSD Jails vs VMWare Server. However, the end result is similar enough to invite comparisons.

Sounds about right

[Just Some Guy]

We use jails a lot at my work. We have a few pretty beefy "jail servers", and use FreeBSD's ezjail port to manage as many instances as we need. Need a new spamfilter, say? sudo ezjail-admin create spam1.example.com 192.168.0.5 and wait for 3 seconds while it creates a brand new empty system. It uses FreeBSD's "nullfs" filesystem to mount a partially populated base system read-only, so your actual jail directly only contains the files that you'd install on top of a new system. This saves drive space, makes it trivially easy to upgrade the OS image on all jails at once (sudo ezjail-admin update -i), and saves RAM because each jail shares the same copy of all the base system's shared libraries.

For extra fun, park each jail on its own ZFS filesystem and take a snapshot of the whole system before doing major upgrades. Want to migrate a jail onto a different server? Use zfs send and zfs receive to move the jail directory onto the other machine and start it.

The regular FreeBSD 7.2 jails already support multiple IP addresses and any combination of IPv4 and IPv6, and each jail can have its own routing table. FreeBSD 8-CURRENT jails also get their own firewall if I understand correctly. You could conceivably have each jail server host its own firewall server that protects and NATs all of the other images on that host. Imagine one machine running 20 services, all totally isolated and each running on an IP not routable outside of the machine itself - with no performance penalty.

Jails might not be the solution to every problem (you can't virtualize Windows this way, although quite a few Linux distros should run perfectly), but it's astoundingly good at the problems it does address. Now that I'm thoroughly spoiled, I'd never want to virtualize Unix any other way.

Re:Sounds about right

[drmofe]

Someone want to compare and contrast FreeBSD jails with openbsd + sysjail?

Re:Sounds about right

[Anonymous Coward]

Agree 100%. We are a BSD shop, and we have been enjoying jails for quite a while. Sandboxing, virtualizing, security advantages. It's great!

Re:Sounds about right

[d3matt]

Just as a curiosity... Have you guys ever used jails for cross-compiles similar to scratchbox?

Re:Sounds about right

[Just Some Guy]

I'm not too up on sysjail, but it looks like it's implemented on top of systrace while jails are explicitly coded into the kernel. That probably made sysjail easier to write, but the FreeBSD work has paid off now that they're starting to virtualize the whole network stack so that each jail can have its own firewall and routing.

More to the point: the sysjail project is no longer maintained.

Re:Sounds about right

[Just Some Guy]

I haven't. After looking at that, I'm not sure what you have in mind. Explain a bit and maybe I can help.

Re:Sounds about right

[larry bagina]

sysjail is vulnerable to race conditions

Re:Sounds about right

[Just Some Guy]

Oh, I forgot to mention another much-loved jail use: giving applications their own customized execution environment. Suppose you have some legacy app that requires, say, some ancient version of Perl and a database connector from 1998. Jails are a great way to sandbox that crufty old environment without forcing those limitations onto the rest of your apps.

Re:Sounds about right

[Anonymous Coward]

look how cool i am i posted command line examples!11!!!

Re:Sounds about right

[busstop]

imagine a beowolf cluster of these... ... ooops - wrong decade - sorry!

Re:Sounds about right

[d3matt]

The major problem is that a lot of GNU software is horribly a pain to cross-compile, especially if the project is less mature. The main goal of scratchbox is to hide the fact that your cross-compiling and to only expose the headers and libraries you need to compile in the "correct" place so the Make/autoconf/automake generated files don't have to be massaged too much. Part of this "hiding" is to also disguise the "arch" of the host machine as well, and it almost seems that jails would be equally suited to do this part.

Re:Sounds about right

[sanqui]

In what sense would "quite a few Linux distros [...] run perfectly" in this context?

Re:Sounds about right

[Just Some Guy]

In what sense would "quite a few Linux distros [...] run perfectly" in this context?

In the sense that FreeBSD has excellent Linux binary compatibility and one time I fired off a jail pointed at the root directory of a Gentoo stage3 archive. It didn't work 100% correctly because Gentoo's init scripts aren't exactly designed for such things, but I could SSH into it and play around.

Re:Sounds about right

[sanqui]

In the sense that FreeBSD has excellent Linux binary compatibility and one time I fired off a jail pointed at the root directory of a Gentoo stage3 archive. It didn't work 100% correctly because Gentoo's init scripts aren't exactly designed for such things, but I could SSH into it and play around.

Wow. That's seriously impressive - I knew about the binary compatibility, but didn't expect it would bootstrap a Linux distro.

Re:Sounds about right

[Just Some Guy]

Like I said, it wasn't perfect. Still, FreeBSD installs a version of Linux in a subdirectory (Fedora Core 6 is the current default, I think) to provide shared libraries for apps that need them. Since FreeBSD doesn't have native flash support, people wanting to watch Youtube on a FreeBSD desktop are often advised to install Firefox for Linux.

What's the diff between jail and zone?

[Vip]

FTA, "Jails are a sort of lightweight virtualization technique available on the FreeBSD platform. They are like a chroot environment on steroids where not only the file system is isolated out but individual processes are confined to a virtual environment - like a virtual machine without the machine part."

Not knowing much about FreeBSD and it's complementary software, what is the difference between FreeBSD Jail and Solaris Zones?
A Solaris Zone could also be described the same way.

Vip

Re:What's the diff between jail and zone?

[jra]

They sound a fair amount like what I understand OpenVZ to be about as well; does the comparison hold there, too?

free beats fee most of the time

[xzvf]

This is slightly off the server virtualization topic, but I had a similar experience with LTSP and some costly competitors. Using LTSP we were able to put up 5X the number of stable Linux desktops on the same hardware. I'd tell every organization out there to do a pilot bake-off as often as possible. It won't happen all the time, but I suspect that more often than not, the free open solution, properly setup will beat the slickly marketed, closed proprietary solution.

Re:free beats fee most of the time

[HateBreeder]

Great... but what's LTSP?

Why do sysadmins assume that everyone else is also a sysadmin who bothers to memorize all these stupid acronyms?

Sure, I googled it, and I hope you meant "Linux Terminal Server Project". But Why not just say so immediately?! Most people won't bother listening to what you have to say if they need too use a search engine to figure out key pieces of information just to understand the context of your words!

Re:free beats fee most of the time

[jdfox]

True, he might have been talking about the Lutheran Theological Seminary at Philadelphia . There is a certain ambiguity there. :-)

Welcome to Slashdot, "News for Nerds". You may find that its readers tend to use lots of initialisms, acronyms and computer slang, especially when discussing computing issues. If you like everything spelled out and linked for you, then you might prefer to read CNET instead.

BTW (by the way), CNET doesn't appear to stand for anything but CNET. :-)

Re:free beats fee most of the time

[Colin Smith]

This is a feature of Unix/linux memory management. Now... If you were to separate out your applications and run each on it's own server (particularly the big bloated apps), you would be able to load the servers even more highly still, and the apps will run faster because more of their code will be shared between users and more will be resident in the cpu caches. e.g. Have an openoffice server or cluster, have a firefox server or cluster. Use something like gridengine to run jobs on the cluster you want.

Re:free beats fee most of the time

[selven]

I don't know, you should go set your CSS and SMTP preferences and the SLA servers should give you the correct BDSM.

Re:free beats fee most of the time

[pbhj]

It's a barrier to entry .. if you know what LTSP is then the post might be relevant, if not then it certainly won't.

How had you not heard of LTSP?

Re:free beats fee most of the time

[jdfox]

I'm curious: if you're not interested in something as "low end" as systems administration, then why would you be interested in a Slashdot discussion on VMware and BSD jails? :-)

And nobody's asking you to memorize what LTSP stands for. Just double-click the text in Firefox, right-click and choose search. So much quicker and more effective than asking everyone to spell out abbreviations. It's a win-win!

Re:free beats fee most of the time

[Anonymous Coward]

Because IO Performance of a virtualized server and the effectiveness of this "jails" method as an alternative, is an interesting topic.

Much more so than a terminal server...

Besides, most people in technology have heard about vmware and everyone knows what BSD is... LTSP? not so much.

Re:free beats fee most of the time

[jdfox]

BSD? Never heard of it. Netcraft confirms it.

Re:free beats fee most of the time

[Anonymous Coward]

... you might prefer to read CNET instead.

Ooooh scorch of the burning flame,
a heavenly sea of napalm raining alight upon thy prey,
roasting flesh off bones, burning to ages eternal.

BURN!

Re:free beats fee most of the time

[swillden]

So why should I bother memorizing field specific acronyms for projects like LTSP that have little or no interest to me?

No reason. But you shouldn't really be surprised when they crop up in field-specific discussions. Would you expect participants in a discussion about board design to spell out ASIC and FPGA?

Re:free beats fee most of the time

[Just Some Guy]

I hate to sound condescending.. but system administration is considered the lower end of the technology community.

You don't sound condescending; you sound ignorant. Routine system maintenance is low end, getting to play with new (to commodity hardware) virtualization techniques and ZFS and SANs and HA systems isn't quite the same as staring blankly at a re-purposed desktop.

Put another way: it's cool that you like writing drivers, but if they suck, I'm the one who gets to blackball your company on purchase orders.

Re:free beats fee most of the time

[Anonymous Coward]

Right, well my BLT drive on my computer just went AWOL, and I've got this big project due tomorrow for Mr. Kawasaki, and if I don't get it in, he's gonna ask me to commit Hari Kari...

Re:free beats fee most of the time

[jra]

Funny goes to 6. Who knew?

Re:free beats fee most of the time

[Anonymous Coward]

Hahahahaha! You pwned that non-nerd! Uh oh! If he can't google an acronym I supposed he is s.o.l. for 1334 5p34k!

Re:free beats fee most of the time

[jcrousedotcom]

I might argue that just a bit. I am familiar with linux and have a decent handle on administrating a linux box but am mainly in the Windows World. This will start a bunch of snarky comments regarding how superior Linux is vs. Windows, and to that point, I will say, we run Linux here as well - in the form of Novell. :)

Seriously though - not everyone is familiar with all abreviations for all services on all OS's. If I really cared what the the OP was talking about, I'd look it up but half the time, unless I am really interested or it is something that I think I should really know what it is, I just gloss over / skip the comment....

Notwithstanding - we all should endeavour to broaden our field of knowledge.

Re:free beats fee most of the time

[Anonymous Coward]

Put another way: it's cool that you like writing drivers, but if they suck, I'm the one who gets to blackball your company on purchase orders.

The one who sounds ignorant is you. He wasn't talking about pedestrian systems like ZFS and SANS and HA systems. He wasn't talking about writing drivers.

How do you think weather forecasting is done? With a driver? How about data mining or machine learning? These special purpose systems have specific needs, such as fast IO, large CPU caches, lots of RAM, and so on. Deploying these systems is a serious challenge, since the implementation's performance needs to be considered during the design phase.

Re:free beats fee most of the time

[ccady]

When I see a term or acronym on Slashdot that don't know about, I go look up the item and learn something. I am often glad that I do. Except that goatse thing.

Re:free beats fee most of the time

[Anonymous Coward]

Hahahahaha! Except he's not a non-nerd! He writes device drivers for a living! Hahahahaha!

Re:free beats fee most of the time

[Just Some Guy]

He wasn't talking about writing drivers.

Quote: "i do logic design and also write low level software, mostly linux device drivers for our hardware [...] I hate to sound condescending.. but system administration is considered the lower end of the technology community. Technology creators (i.e. R&D) is the high end, whereas maintenance (i.e. sysadmins) is the low end."

So yeah, he was. He's a driver writer and a lot higher ranking than us lowly sysadmins.

Deploying these systems is a serious challenge, since the implementation's performance needs to be considered during the design phase.

So you agree with me that system administration is a challenging and rewarding occupation.

Re:free beats fee most of the time

[Anonymous Coward]

Try this too: http://simple.wikipedia.org/wiki/Linux

Re:free beats fee most of the time

[Have Brain Will Rent]

The grandparent has a completely valid point. It doesn't matter if one is a geek or not, one simply can't know all the acronyms. In writing there is a common practice which is to spell out the meaning of an acronym at the point of first use in a text. Of course that assumes that the writer is interested in communicating as effectively as possible to as large an audience as possible. If you don't support that goal then sure, go ahead, deride the person who asks for clearer communication... obviously there is some way in which it is making you feel good to do that.

Re:free beats fee most of the time

[Anonymous Coward]

Would you like us to explain what the VM in VMware is also?? I see that SQLite was mentioned in the article to. Would you liked that spelled out also? How about what BSD stands for? We wouldn't want you to get confused or something.

AFIK (As Far As I Know) This site is primarily focused to People of the administrative level of things (Hence the name of the site). If you want to hang out with the big boys, then bring a dictionary. If you want to be a part of the geek/nerd environment then get used to it. We throw Acronyms like WWE wrestlers throw matches.

Re:free beats fee most of the time

[Anonymous Coward]

It's a win-win!

Not if somebody else comes up with a tech-related project that uses the same acronym. So what does CSS mean again? Or CGI? And how is it more efficient to make everyone have to figure out what you meant instead of just telling them?

Re:free beats fee most of the time

[dickens]

Boy, this seems to have troll written all over it. But still I get sucked in. System administration may be a slightly wider world than you suspect, embracing, for the SMB, support strategy, resource planning and *then* technology selection. Bring in a dose of network engineering, traditional IS support, and tie it all with a bow of security policy and disaster recovery planning.

Oh, and the users still just see Windows, and don't notice much in the way of change.

BTW: SMB in this context means Small and Medium Business, not Server Message Block, and IS means Information Systems, like um, accounting and stuff, so we get paid?

Re:free beats fee most of the time

[rant64]

Would you expect participants in a discussion about board design to spell out ASIC and FPGA?

No, but xzvf admitted himself that he was posting slightly off-topic and therefore should not expect that everyone is familiar with it.
LTSP is not the topic at hand, virtualization is. I agree with HateBreeder.

Re:free beats fee most of the time

[obscuro]

But Why not just say so immediately?! Most people won't bother listening to what you have to say if they need too use a search engine to figure out key pieces of information just to understand the context of your words!

See top search result: http://www.google.com/search?rlz=1C1CHMC_enUS291US305&sourceid=chrome&ie=UTF-8&q=ltsp

Slashdot would probably begin to suck if people followed this kind of full service philosophy on every post. The posts would be longer, there would be information presented within the posts that a large number of users already know (or could find out with less than 5 SECONDS of effort). I, for one, don't read slashdot for some newsy eye massage with release.

BTW - Given the readership numbers for slashdot, I think we've proven that most people HERE do listen even though we sometimes need a search engine. I also have a dictionary on my shelf....

Re:free beats fee most of the time

[swillden]

LTSP is not the topic at hand, virtualization is. I agree with HateBreeder.

Slightly off the topic, perhaps, but both are within the sphere of enterprise system administration, and any competent sysadmin would know of both. Plus, LTSP isn't a particularly obscure topic within that realm. I'm not a system administrator on anything but my own computer, and I know about it.

I don't know enough about hardware design to come up with a truly accurate analogy, but I'm sure there are plenty.

Re:free beats fee most of the time

[rant64]

I don't want an argument over terms here, but I think your generalization of a competent sysadmin does not fit in here. I can really throw in a handful of acronyms of which I think a sysadmin should be aware of in my line of work that you don't, and I'm sure you can do the same for whatever field you're specialized in. But that wouldn't make either of us incompetent sysadmins. (btw my job description doesn't say sysadmin either).

The point is: many readers here who might be interested in the Linux Terminal Server Project don't give the GGGGGP's post a second thought because they don't recognize the acronym - which from my POV is nowhere near to being ubiquitous technology like virtualization, and if people don't pick up hints like this never will be.

Re:free beats fee most of the time

[Mistoffeles]

Seems you missed the relevant point, which is *which* systems he was referring to when he said "Designing these systems is a serious challenge."

Out of context arguments are fail in any field, much less IT. There are a lot more Systems Administrators around (indeed, we are a dime a dozen in this day of cert mill tech colleges) than tightly focused experts who perform a very specific function (and get paid much higher salaries for it, because there are relatively few people capable of performing these functions).

Not surprising

[Anonymous Coward]

Virtual machines tend to be fast in theory, but slow in practice. Just look at Java.

Government IT is being poisoned by virtualization

[kriston]

The new buzzword of Virtualization has reached all corners of the US Government IT realm. Blinded by the marketing hype of "consolidation" and "power savings" agencies of the three-letter variety are falling over themselves awarding contracts to "virtualize" the infrastucture. Cross-domain security be damned, VMWare and Microsoft SoftGrid Hyper-v Softricity Whatevers will solve all their problems and help us go green at the very same time, for every application, in every environment, for no reason.

This is the recovery from the client-server binge-and-purge of the 1990s.

Here we go again.

Virtualization != Performance

[gmuslera]

If you really need all the performance you can get for a service, don't virtualize it, or at least check that what you can get is enough, Virtualization have a lot of advantages, but dont give you the full resources of the real machine is running into (and if well how much you lose depend on the kind of virtualization you use, still wont be full). Maybe the 10x number could be VMWare fault or just a reasonable consequence of how is doing virtualization (maybe taking into account disk IO performance you could explain a good percent of that number).

Solaris Zones also

[ltmon]

Zones are the same concept, with the same benefit.

An added advantage Solaris zones have is flavoured zones: Make a Solaris 9 zone on a Solaris 10 host, a Linux zone on a Solaris 10 host and soon a Solaris 10 zone on an OpenSolaris host.

This has turned out much more stable, easy and simply effecient than our Vmware servers, which we now only have for Windows and other random OS's.

Re:Solaris Zones also

[Anonymous Coward]

Zones are just the operating system partitioned, so it doesn't make sense to run linux in a zone. You can however, run a linux branded zone, which emulates a linux environment, but it's not the same as running linux in a zone. It's running linux apps in solaris.

LDOMS are hardware virtualization, so you can run Linux in them. Only some servers are supported, though.

Just thought i better clarify.

Is this a surprise?

[diamondsw]

Amazing! Not running several additional copies of an operating system with all of the needless overhead involved is faster! Who would have guessed?

Sometimes a virtual machine is far more "solution" than you need. If you really want the same OS with lots of separated services and resource management... then run a single copy of the OS and implement some resource management. Jails are just one example - I find Solaris Containers to be much more elegant. Of course, then you have to be running Solaris...

Re:Is this a surprise?

[QuoteMstr]

Or Linux containers for that matter.

(Or for something more mature today, but implemented as a large out-of-tree patch, OpenVZ)

XenServer from Citrix -- eewww

[xzvf]

XenServer is a great product and has many skilled developers. The "from Citrix" really gives me a queasy feeling. I know the products are solid and innovative, but so many people I hear out in the wild, scream and run from Citrix. It might be behind the reason Ubuntu and Red Hat are backing KVM for virtualization. Even to the point where RH bought Qumarant (KVM "owners").

Re:XenServer from Citrix -- eewww

[gbr]

I have to agree, the 'from Citrix' makes me queasy for a couple of reasons.

1. I've had issues with Citrix products in the past
2. Xen is the work of many people, not just Citrix.

Issue 2 compensated for issue 1, and it was further assuaged by the performance of the VM's. Very nice. I was also nice that Citrix made XenServer free just as we were about to write a check.

Re:XenServer from Citrix -- eewww

[Anonymous Coward]

>The "from Citrix" really gives me a queasy feeling. . . so many people I hear out in the wild, scream and run from Citrix.

That's because it runs on a completely unsuitable substrate for multi-user, server-side computing: Windoze.

Different tools for different jobs

[ErMaC]

So I would love to RTFA to make sure about this, but their high-performance web servers running on FreeBSD jails are down, so I can't...

But here's what I do know. FreeBSD hasn't been a supported OS on ESX Server until vSphere came out less than two weeks ago. That means that either:
A) They were running on the Hosted VMware Server product, whose performance is NOT that impressive (it is a Hosted Virtualization product, not a true Hypervisor)
or B) They were running the unsupported OS on ESX Server, which means there was no VMware Tools available. The drivers included in the Tools package vastly improve things like storage and network performance, which means no wonder their performance stunk.

But moreover, Jails (and other OS-virtualization schemes) are different tools entirely - comparing them to VMware is an apples-to-oranges comparison. Parallels Virtuozzo would be a much more apt comparison.

OS-Virtualization has some performance advantages, for sure. But do you want to run Windows and Linux on the same physical server? Sorry, no luck there, you're virtualizing the OS, not virtual machines. Do you want some of the features like live migration, high availability, and now features like Fault Tolerance? Those don't exist yet. I'm sure they will one day, but today they don't, or at least not with the same level of support that VMware has (or Citrix, Oracle or MS).

If you're a company that's trying to do web hosting, or run lots of very very similar systems that do the same, performance-centric task, then yes! OS Virtualization is for you! If you're like 95% of datacenters out there that have mixed workloads, mixed OS versions, and require deep features that are provided from a real system-level virtualization platform, use those.

Disclosure: I work for a VMware and Microsoft reseller, but I also run Parallels Virtuozzo in our lab, where it does an excellent job of OS-Virtualization on Itanium for multiple SQL servers...

Re:Different tools for different jobs

[Anonymous Coward]

But here's what I do know. FreeBSD hasn't been a supported OS on ESX Server until vSphere came out less than two weeks ago.

Really? VMware tools for freebsd have been available for years. You can even run them on openbsd (with freebsd compatibility mode enabled).

There's even this slashdot story from 2004 about freebsd 4.9 being supported as an esx guest.

Re:Different tools for different jobs

[mevets]

|There's even this slashdot story from 2004 about freebsd 4.9 being supported as an esx guest.

Yes, but that was before bsd was confirmed dead.

Re:Different tools for different jobs

[Anonymous Coward]

"So I would love to RTFA to make sure about this, but their high-performance web servers running on FreeBSD jails are down, so I can't..."

No chance that all the bandwidth they have is being sucked up?

"FreeBSD hasn't been a supported OS on ESX Server."

Not applicable since when I RTFA it said they were running VMware Server 1.

Re:Different tools for different jobs

[Just Some Guy]

If you're a company that's trying to do web hosting, or run lots of very very similar systems that do the same, performance-centric task, then yes! OS Virtualization is for you! If you're like 95% of datacenters out there that have mixed workloads, mixed OS versions, and require deep features that are provided from a real system-level virtualization platform, use those.

If only it weren't mathematically impossible to mix technologies in the datacenter so that you could run jails and VMware in the same building and divide tasks amongst them as appropriate, but alas.

OpenVZ & Virtuozzo are my favorite way to go

[pyite69]

I would expect that the BSD product is similar in design - basically chroot on steroids.

One runs on Solaris, one runs on BSD

[_merlin]

FreeBSD Jails are the same thing as Solais Zones, just on FreeBSD. Since FreeBSD is about evil daemons, they need an evil-sounding marketing name for it. More seriously, they probably just didn't want to bring on the wrath of lawyers for trademark infringement.

Re:One runs on Solaris, one runs on BSD

[jbellis]

> they probably just didn't want to bring on the wrath of lawyers for trademark infringement.

FreeBSD jails predate Solaris zones by five years.

Re:One runs on Solaris, one runs on BSD

[Anonymous Coward]

And the generic term for this kind of virtualization is "containers" (and they should be used as much as possible, ie as long as you only need one and only kernel, ie 99% of the time; not to speak about such containers-capable kernel can be virtualized xen-style and such).

You can also find them through VServer and OpenVZ, on Linux. And it rocks! Definately!

Re:One runs on Solaris, one runs on BSD

[Capt James McCarthy]

> they probably just didn't want to bring on the wrath of lawyers for trademark infringement.

FreeBSD jails predate Solaris zones by five years.

And soon they will be called Soracle Meditation Gardens.

Re:One runs on Solaris, one runs on BSD

[Anonymous Coward]

> they probably just didn't want to bring on the wrath of lawyers for trademark infringement.

FreeBSD jails predate Solaris zones by five years.

When has that ever stopped a lawyer from claiming the reverse?

Re:One runs on Solaris, one runs on BSD

[rbanffy]

And, IIRC, Solaris zones allow you to run multiple, different Solaris versions.

I/O on the free "VMWare Server" sucks

[mrbill]

The I/O performance on the free "VMWare Server" product *sucks* - because it's running on top of a host OS, and not on the bare metal.
I'm not surprised that FreeBSD Jails had better performance. VMWare Server is great for test environments and such, but I wouldn't ever use it in production.
It's not at all near the same class of product as the VMWare Infrastructure stuff (ESX, ESXi, etc.)

VMWare offers VMWare ESXi as a free download, and I/O performance under it would have been orders of magnitude better.
However, it does have the drawback of requiring a Windows machine (or a Windows VM) to run the VMWare Infrastructure management client.

Re:I/O on the free "VMWare Server" sucks

[zonky]

ESXi does also have many limitations around supported hardware. That said, there are some good resources around running ESXi on 'white box' hardware.

http://www.vm-help.com//esx40i/esx40_whitebox_HCL.php

Re:I/O on the free "VMWare Server" sucks

[snookums]

There's overhead, but not 10x worse performance unless you're hitting the disk far more in the VM than you were in the native deployment.

The "gotcha" is that VMWare Server will, by default, use file-backed memory for your VMs so that you can get in a situation where the VM is "thrashing", but neither the host nor guest operating system shows any swap activity. The tell-tale sign is that a vmstat on the host OS will show massive numbers of buffered input and output blocks (i.e. disk activity) when you're doing things in the VM which should not require this amount of disk troughput.

A possible solution is:

1. Move the backing file to tmpfs*
2. Increase your mounted tmpfsto cover most of the host machine RAM (I'd say total RAM - 1 GB).
3. Allocate RAM to your VMs in such a way that you are not over-committed (total of all VMs not more than tmpfs size set at step 2).

*Take a look at the option mainMem.useNamedFile = "FALSE"

Virtualization doesn't make sense

[QuoteMstr]

Well, in one case it does: when you're trying to run a different operating system simultaneously on the same machine. But in most "enterprise" scenarios, you just want to set up several isolated environments on the same machine, all running the same operating system. In that case, virtualization is absofuckinglutely insane.

Operating systems have been multi-user for a long, long time now. The original use case for Unix involved several users sharing a large box. Embedded in the unix design is 30 years of experience in allowing multiple users to share a machine --- so why throw that away and virtualize the whole operating system anyway?

Hypervisors have become more and more complex, and a plethora of APIs for virtualization-aware guests has appeared. We're reinventing the kernel-userland split, and for no good reason.

Technically, virtualizaiton is insane for a number of reasons:

• Each guest needs its own kernel, so you need to allocate memory and disk space for all these kernels that are in fact identical
• TLB flushes kill performance. Recent x86 CPUs address the problem to some degree, but it's still a problem.
• A guest's filesystem is on a virtual block device, so it's hard to get at it without running some kind of fileserver on the guest
• Memory management is an absolute clusterfuck. From the point of view of the host, each guest's memory is an opaque blob, and from the point of view of the guest, it has the machine to itself. This mutual myopia renders the usual page-cache algorithms absolutely useless. Each guest blithely performs memory management and caching on its own resulting in severely suboptimal decisions being made.

  In having to set aside memory for each guest, we're returning to the OS9 memory mangement model. Not only are we reinventing the wheel, but we're reinventing a square one covered in jelly.

FreeBSD's jails make a whole lot of sense. They allow several users to have their own userland while running under the same kenrel --- which vastly improves, well, pretty much everything. Linux's containers will eventually provide even better support.

Re:Virtualization doesn't make sense

[MichaelSmith]

If you are going to hire cheap MCSEs to manage all your systems, including the unix ones then it makes sense to be able to put those unix systems inside a little box on your screen with nice borders around it so you can easily see what connects to what.

Saving money on hardware will just cost you kickbacks from the supplier anyway. There is no advantage in that.

Re:Virtualization doesn't make sense

[Anonymous Coward]

Saving money on hardware will just cost you kickbacks from the supplier anyway. There is no advantage in that.

If you honestly believe that, I have some $10 off your next $1,000,000 purchase coupons I'd like to send you...

Re:Virtualization doesn't make sense

[syousef]

Virtualization DOES make sense, when you're trying to solve the right problem. Do not blame the tool for the incompetence of those using it. It's no good using a screwdriver to shovel dirt and then blaming the screwdriver.

Virtualization is good for many things:
- Low performance apps. Install once, run many copies
- Excellent for multiple test environments where tests are not hardware dependant
- Infrequently used environments, like dev environments, especially where the alternate solution is to provide physical access to multiple machines
- Demos and teaching where multiple operating systems are required
- Running small apps that don't run on your OS of choice infrequently

Virtualization is NOT good for:
- High performance applications
- Performance test envrionemnts
- Removing all dependence on physical hardware
- Moving your entire business to

Your specific concerns:
# Each guest needs its own kernel, so you need to allocate memory and disk space for all these kernels that are in fact identical

Actually this depends on your virtualization solution

# TLB flushes kill performance. Recent x86 CPUs address the problem to some degree, but it's still a problem.

So is hard disk access from multiple virtual operating systems contending for the same disk (unless you're going to have one disk per guest OS...even then are you going through one controller?) Resource contention is a trade-off. If all your systems are going to be running flat out simultaneously virtualization is a bad solution.

# A guest's filesystem is on a virtual block device, so it's hard to get at it without running some kind of fileserver on the guest

You can often mount the virtual disks in a HOST OS. No different to needing software to access multiple partitions. As long as the software is available, it's not as big an issue.

# Memory management is an absolute clusterfuck. From the point of view of the host, each guest's memory is an opaque blob, and from the point of view of the guest, it has the machine to itself. This mutual myopia renders the usual page-cache algorithms absolutely useless. Each guest blithely performs memory management and caching on its own resulting in severely suboptimal decisions being made

A lot of operating systems are becoming virtualization aware, and can be scheduled cooperatively to some degree. That doesn't mean your concern isn't valid, but there is hope that the problems will be reduced. However once again if all your virtual environments are running flat out, you're using virtualization for the wrong thing.

Re:Virtualization doesn't make sense

[QuoteMstr]

A lot of operating systems are becoming virtualization aware

Which ends up being as complex as the kernel-userland boundary, so why not just use a kernel-userspace boundary in the first place?

Re:Virtualization doesn't make sense

[billybob_jcv]

Sorry, but I think you're missing several important points. In a company with several hundred physical servers and limited human resources, no one has the time to fool around with tuning a kernal and several apps to all run together in the same OS instance. We need to build standard images and deploy them very quickly, and then we need a way to easily manage all of the applications. We also need to be able to very quickly move applications to different HW when they grow beyond their current resources, we refresh server HW or there is a HW failure. High Availability is expensive, and it is just not feasible for many midrange applications that are running on physical boxes. Does all of this lead to less than optimal memory & I/O performance? Sure - but if my choice is hiring 2 more high-priced server engineers, or buying a pile of blades and ESX licenses, I will bet buying more HW & SW will end up being the better overall solution.

Re:Virtualization doesn't make sense

[Anonymous Coward]

"Each guest needs its own kernel, so you need to allocate memory and disk space for all these kernels that are in fact identical"

Wrong - transparent page sharing and linked cloning address both of these "problems," which BTW also exist in a physical world. Keeping the kernels separate is a good thing when dealing with the typical shit applications that get installed in the average datacenter. (Yes, I know TPS and linked clones are only available on one product.)

"TLB flushes kill performance. Recent x86 CPUs address the problem to some degree, but it's still a problem."

Wrong - Hardware virtualization (AMD-V and Intel VT) address this nicely. (And also paravirt to a lesser extent.)

"A guest's filesystem is on a virtual block device, so it's hard to get at it without running some kind of fileserver on the guest"

WTF are you even talking about there? Get at it from where?

"From the point of view of the host, each guest's memory is an opaque blob, and from the point of view of the guest, it has the machine to itself."

Wrong - tools installed in the guest give the host a window into the VM, which the hypervisor can use to make smart decisions about memory allocation.

"FreeBSD's jails make a whole lot of sense."

Maybe for FreeBSD apps, but what percentage of datacenter apps run on FreeBSD? Maybe 10 percent? (Probably far less.)

"Operating systems have been multi-user for a long, long time now. The original use case for Unix involved several users sharing a large box. Embedded in the unix design is 30 years of experience in allowing multiple users to share a machine --- so why throw that away and virtualize the whole operating system anyway?"

Virtualization is not about users sharing the box, it's about applications co-existing on the box, even if those applications require 50 different operating systems. Jails and virtualization solve very different problems. Besides, nobody says that you can't use jails where appropriate and virtualization where appropriate.

Re:Virtualization doesn't make sense

[QuoteMstr]

How the fuck is someone with (only) an MCSE supposed to manage a Unix system?

Re:Virtualization doesn't make sense

[QuoteMstr]

Wrong - transparent page sharing and linked cloning address both of these "problems,"

Inefficiently as fuck, by the way

Keeping the kernels separate is a good thing when dealing with the typical shit applications

Uh, why? Even shit applications don't replace or extend the kernel

Maybe for FreeBSD apps

FreeBSd runs Linux apps just fine last time I checked

Re:Virtualization doesn't make sense

[MichaelSmith]

How the fuck is someone with (only) an MCSE supposed to manage a Unix system?

Easy.

Re:Virtualization doesn't make sense

[SlothDead]

Well, no. VMware virtual machines share the memory that is identical, so you don't need to allocate that memory for every client (e.g. Kernel). Also, they share the files on the hard drive as long as they are identical and when running the same programs they even share some chunks of the RAM used by that program.

Re:Virtualization doesn't make sense

[QuoteMstr]

VMware virtual machines share the memory that is identical

Inefficient as fuck. Whereas if they'd just been processes running under the same OS, the kernel would already know they were sharing the same page.

Re:Virtualization doesn't make sense

[rabbit994]

Easy, we replace it with Windows.

I only partly kid, in most cases, if all you have are Windows Admins, it makes good sense to replace Unix stuff with Windows if equivalent software is available and price isn't horribly cost prohibitive.

Re:Virtualization doesn't make sense

[AcidPenguin9873]

Your points are all valid, but they are some of the areas that virtualization systems have addressed in the past 10 years (or longer if you were running an IBM system).

Each guest needs its own kernel, so you need to allocate memory and disk space for all these kernels that are in fact identical

I'm pretty sure VMWare can detect when the same block of the same file is mapped into multiple guests, and share the physical page. Plus, the kernel's memory image is small compared to, say, the database server you're running on it. I guess there's overhead like an extra set of page tables (either nested page tables managed by the guest, or shadow page tables managed by the host). Overall a small effect I think.

TLB flushes kill performance. Recent x86 CPUs address the problem to some degree, but it's still a problem.

Any context switch between two userspace programs in a non-virtualized system needs a TLB flush too (BSD jails included). Or, if you're using a processor that has a tagged TLB, you don't need to flush it, but your virtualized guest gets the no-TLB-flush benefit too.

A guest's filesystem is on a virtual block device, so it's hard to get at it without running some kind of fileserver on the guest

Again I don't think this is a huge deal. Aren't there drivers to allow a host to see inside a guest's block device and/or filesystem?

Memory management is an absolute clusterfuck.

In a naive hypervisor, yes. In more mature hypervisors, not really. See the following articles for solutions on fully virtualized and paravirtualized guests, respectively: http://www.usenix.org/events/osdi02/tech/waldspurger/waldspurger_html/node6.html
http://lwn.net/Articles/198380/

Re:Virtualization doesn't make sense

[chez69]

Right now your looking at it from a completely x86 view. Look at it from teh point of view from a hardware based system that's been doing it for years. A lot of these problems have been solved already.

Let's say you have a lot of servers at big corp. Each runs a specialized application, and each application is required to be isolated from the rest. A good VM system like zVM can help you a ton. You get a hardware platform that has tons of mature disaster recovery solutions, and a hypervisor that can dynamically allocate resorces between different VMs to the point where you don't even see it.

I mention zVM a lot because I know a lot of folks that are involved with large scale rollouts of it, in production, with great results.

The downside is that you need people who know what they're doing, and the hardware is expensive as hell.

Re:Virtualization doesn't make sense

[chez69]

isolation? Ring 0 bugs would kill all the jails in that kernel, right?

Don't think of VMware as the right model, think about VM (maiframe OS) instead.

Re:Virtualization doesn't make sense

[miffo.swe]

If you have admins that cant manage anything but Windows you should really ponder replacing them. Admins that cant wrap their head around anything but windows arent really that good at windows either. I can manage just about anything and i have seen collegues spend insane amount of money on stuff even a simple script or even a .bat file could fix. The right tool for the right job requires people to know a bit more than an MCSE gives.

Those people are also often the ones who stalls upgrades for all its worth because they dont want to learn new stuff.

Re:Virtualization doesn't make sense

[ion.simon.c]

It seems to me that jail-busting bugs in BSD's kernel are much less likely than sandbox-busting bugs in VMWare's software.

Re:Virtualization doesn't make sense

[afidel]

Guess you don't live in the real world of IT where DLL/library/RPM/Deb hell is a way of life and the effort in testing changes on a box containing N services is ~(N!) Also many vendor's won't support multiple pieces of their solution running on the same OS install. Not to mention the fact that security is never absolute and having a harder boundary between services can be a good thing. Oh and VMWare solves the multiple kernel problem by using page sharing, but it does come at a small cost in CPU power. There are many thousands of companies saving real money and real watts by using virtualization, it might not be the ultimate solution for every problem but they sure are a good way to EASILY take better advantage of modern hardware.

Re:Virtualization doesn't make sense

[Macka]

# Each guest needs its own kernel, so you need to allocate memory and disk space for all these kernels that are in fact identical

Actually this depends on your virtualization solution

No it doesn't. The parent is clearly talking/complaining about VMware, Xen, Kvm type virtualization, and guest OS instances for all those require their own kernel. He isn't talking about jails/container solutions (FreeBSD Jails, OpenVZ, Solaris Containers, etc) or none of his points would make any sense.

# A guest's filesystem is on a virtual block device, so it's hard to get at it without running some kind of fileserver on the guest

You can often mount the virtual disks in a HOST OS. No different to needing software to access multiple partitions. As long as the software is available, it's not as big an issue.

Not without shutting the guest down first. If you mount a filesystem on a disk/partition twice and that filesystem is not a specially designed cluster filesystem, and the two OS instances are not part of the same cluster, then you WILL get data corruption. The parent's point is valid !

You should have stopped at your list of what virtualization is good and not good for. You let yourself down after that.

Re:Virtualization doesn't make sense

[syousef]

No it doesn't. The parent is clearly talking/complaining about VMware, Xen, Kvm type virtualization, and guest OS instances for all those require their own kernel. He isn't talking about jails/container solutions (FreeBSD Jails, OpenVZ, Solaris Containers, etc) or none of his points would make any sense

So the parent is using the wrong kind of virtualisation, then blaming the tool. My screw driver won't shovel dirt very well. Bad screw driver. He found a solution that better fitted what he was trying to do and is implying that VMWare is therefore bad. When you misuse a tool then conclude the tool is bad, it's quite valid to point out there are tools out there better suited to the job.

ot without shutting the guest down first. If you mount a filesystem on a disk/partition twice and that filesystem is not a specially designed cluster filesystem, and the two OS instances are not part of the same cluster, then you WILL get data corruption. The parent's point is valid !

One of us is misreading. I thought the parent was complaining that he couldn't access files on the virtualized OS without starting up his guest. I was pointing out there are solutions to mount virtual partitions. (I've personally only done it on MS Virtual server as a workaround when Vista restore didn't work).

If he wants to access the file system read-write both through guest and host at the same time what he's doing is silly. Any 2 systems accessing the same partition read-write, virtualized or not, will cause problems unless the file system is written specifically to accommodate that. (I'm not aware of any that do off the top of my head).

You should have stopped at your list of what virtualization is good and not good for. You let yourself down after that.

I respectfully disagree.

Re:Virtualization doesn't make sense

[Anonymous Coward]

FreeBSD's jails make sense on paper, but make little sense -- especially from an administration point of view -- when implemented. FreeBSD jails are nothing more than an overglorified chroot.

There are many userland utilities which break horribly with jails.

And the whole "copy /dev and random bullshit from /bin, /usr/libexec, /usr/lib, etc." concept is an absolute disgrace. Good luck keeping all of that managed/updated the next time you build/install world.

Avoid FreeBSD jails. Surely Linux has something like jails which doesn't involve this kind of ancient idiocy.

Re:Virtualization doesn't make sense

[julesh]

Technically, virtualizaiton is insane for a number of reasons:

        * Each guest needs its own kernel, so you need to allocate memory and disk space for all these kernels that are in fact identical

1. Who says the kernels are identical? I can run each application on an environment tailored specifically for it. I have a single server running applications that require W2K3, Linux 2.4 and Linux 2.6 here. I also have WinXP, Solaris and OSX images for our developers' use in testing. I don't think this scenario is as rare as you think; there are a hell of a lot of legacy applications out there that are extremely picky about OS versions they run on. I've come across people still running netware because they need an app that depends on it; I'm pretty sure that could be put on a VM too.

2. AIUI, the hypervisor is able to detect identical memory pages and merge them. Disk space is cheap, so I don't care that I need more of it.

TLB flushes kill performance. Recent x86 CPUs address the problem to some degree, but it's still a problem.

Yes, but this is as much a problem with running multiple processes on the same OS, as you have a TLB flush on each user/kernel transition anyway. Kernel/kernel transitions are rare so the additional performance overhead of virtualisation here is minimal.

A guest's filesystem is on a virtual block device, so it's hard to get at it without running some kind of fileserver on the guest

Yes, this is a slight downside, but the administrative overhead involved here is small. If the guest isn't running, I can easily mount its partition in another VM; if it is running, the cost of configuring a fileserver with admin-only access is minimal.

Memory management is an absolute clusterfuck. From the point of view of the host, each guest's memory is an opaque blob, and from the point of view of the guest, it has the machine to itself. This mutual myopia renders the usual page-cache algorithms absolutely useless. Each guest blithely performs memory management and caching on its own resulting in severely suboptimal decisions being made.

Yes & no. The VM is quite easily able to detect pages that have been mapped from disk (i.e. cache pages) and handle them in a sensible fashion, including merging the caches between different systems. One problem with jails and similar techs is that an I/O intensive process running on one jail can effectively seize control of the entire cache, severely degrading performance of other jails. This doesn't happen with VMs, as their cache is effectively partitioned. But, on a system that's behaving properly this does result in needing more memory to achieve the same performance level, yes.

Re:Virtualization doesn't make sense

[Tony Hoyle]

Actually it's pretty efficient - the physical memory usage of the Windows servers under ESX is under 10% of the 'real' memory because almost the entire OS is shared. If they're running similar tasks it takes a lot of activity to cause that to rise significantly.

Re:Virtualization doesn't make sense

[Salamander]

We're reinventing the kernel-userland split, and for no good reason.

Thank you for saying that. The purpose of a multi-tasking multi-user OS is to allow running multiple applications with full isolation from one another. If we need some other piece of software - like a VM hypervisor - to do that, then the OS has failed in its duty. But wait, some people say, it's not just about multiplexing hardware, it's about migration and HA and deploying system images easily. These are also facilities the OS should be providing. Again, if we need some other piece of software then the OS has failed.

One could argue that we've evolved to a point where the functions of an OS have been separated into two layers. One layer takes care of multiplexing the hardware; the other takes care of providing an API and an environment for things like filesystems. Better still, you get to mix and match instances of each layer. OK, fine. Given the Linux community's traditional attitude toward layering (and microkernels, which this approach also resembles) it's a bit inconsistent, but fine. That interpretation does raise some interesting questions, though. People are putting a lot of thought into where drivers should live, and since some drivers are part of "multiplexing the hardware" then it would make sense for them to live in the hypervisor with a stub in the guest OS - just as is being done, I know. But what about the virtual-memory system? That's also a form of hardware multiplexing, arguably the most important. If virtualization is your primary means of isolating users and applications from one another, why not put practically all of the virtual-memory functionality into the hypervisor and run a faster, simpler single-address-space OS on top of that?

If we're going to delegate operating-system functionality to hypervisors, let's at least think about the implications and develop a coherent model of how the two interact instead of the disorganized and piecemeal approaches we see now.

Re:Virtualization doesn't make sense

[rbanffy]

It was OS7, IIRC, not OS9 ;-)

Re:Virtualization doesn't make sense

[Anonymous Coward]

Not only are we reinventing the wheel, but we're reinventing a square one covered in jelly.

KY Jelly by chance?

As for the database...

[orngjce223]

I think they got /.'d.

I don't think you did your research.

[BagOBones]

If you are separating similar work loads like web apps and databases you are probably better off running them within the same os and database server and separating them via security as the poster realized.

However if you have a variety of services that do not do the same thing you can really benefit from separating them in virtual machines and have them share common hardware.

Virtualization also gives you some amazing fault tolerance options that are consistent across different OS and services that are much easier to manage than individual OS and service clustering options.

Re:I don't think you did your research.

[BagOBones]

After looking more closely at the article it sounds like they where trying to use VMWare Server instead of ESX, which explains a lot. If that was the case they were then carring the overhead of the host OS, VMLayer and the multiple guest OS. Not something you do with high performance apps.

Re:I don't think you did your research.

[Anonymous Coward]

I've used UML, quemu, etc.. and eventually I ditched linux for server level stuff for the sole reason that linux doesn't have BSD jails the way FreeBSD does. (I tried the vserver project, but it was overly convoluted and not supported. Still have a legacy linux box that is "stuck" with vserver code)

I'm glad, as it turns out, FreeBSD has better performance for server level tasks anyway I still use virtualization when I need to run alternate OS's, of course. (I found that when using virtualization, you should give it a virtual disk on tmpfs for virtual swap)

Jails are FAR better in most cases, you can share memory, you don't need to run another instance of the kernel, unlike chroot, processes are hidden from each other. (and most freebsd stuff doesn't need the /proc filesystem so you don't have the escape chroot via proc issue) you can use the unionfs to mount filesystems on and off a running jail (unlike the vserver project)

Plain old chroot doesn't quite compare, jails are an excellent way to isolate logical machines, services and processes with roughly the same overhead.

I'm glad slashdot is finally mentioning them!

Re:I don't think you did your research.

[chez69]

isn't vmware esx just a specialized linux distribution that is built just to run vm's?

Re:I don't think you did your research.

[BagOBones]

Not at really, however linux is used.

Linux boot straps the hyper-visor which then takes over, then a small linux VMClient runs on top of the hyper-visor to provide an api to talk to the hyper-visor which is now managing all the low level stuff. This is a very simplified explanation but the basics is that it does not run on linux, the hyper-visor contains no linux code.

Interesting...

[certain death]

I can see how running multiple processes would make Jail better for *BSD, but if you want to run an entirely different OS in a VM, it just isn't there. That said, I don't think VMware is as awesome as Xen, but Xen has trouble running certain OSes that VMware can run without issue (within reason), so I think they all have their strong areas of coverage.

Coral Cache

[Qubit]

http://www.playingwithwire.com.nyud.net/2009/06/virtual-failure-yippiemove-switches-from-vmware-to-freebsd-jails/ Because otherwise it's hosed.

We have no history

[QuoteMstr]

I hate to link to my own comment, but it seems particularly relevant here.

"Here we go again" indeed. Hell, I wasn't around for the first go-round and I recognize it when I see it.

Guys - We found the iTanic customer

[Mr Thinly Sliced]

Parent poster admits to using iTanic - someone tie his hands to the tree while I call the Vet.

We will tranq him and put him in a zoo. This will mean big things for us, big things. Tours on broadway, my picture on the cover of Time....

Re:Guys - We found the iTanic customer

[rbanffy]

I run HP-UX, you insensitive clod!

The cache doesn't help.

[Animats]

That just gets you a cached version of a page with a link to the actual article. The actual article is more useful.

Re:The cache doesn't help.

[Qubit]

Hmmm... the coral cache is snappy for me; the original link is not even loading yet.

Here's the start of the article, in any case:

Jun 01. Virtual Failure: YippieMove switches from VMware to FreeBSD Jails

Our email transfer service YippieMove is essentially software as a service. The customer pays us to run some custom software on fast machines with a lot of bandwidth. We initially picked VMware virtualization technology for our back-end deployment because we desired to isolate individual runs, to simplify maintenance and to make scaling dead easy. VMware was ultimately proven to be the wrong choice for these requirements.

Ever since the launch over a year ago we used VMware Server 1 for instantiating the YippieMove back-end software. For that year performance was not a huge concern because there were many other things we were prioritizing on for YippieMove â09. Then, towards the end of development we began doing performance work. We switched from a data storage model best described as âoea huge pile of filesâ to a much cleaner sqlite3 design. The reason for this was technical: the email mover process opened so many files at the same time that weâ(TM)d hit various limits on simultaneously open file descriptors. While running sqlite over NFS posed its own set of challenges, they were not as insurmountable as juggling hundreds of thousands of files in a single folder. ...

Virtualization is good enough

[Gothmolly]

I work for $LARGE_US_BANK in the performance and capacity management group, and we constantly see the business side of the house buy servers that end up running at 10-15% utilization. Why? Lots of reasons - the vendor said so, they want "redundancy", they want "failover" and they want "to make sure there's enough". Given the load, if you lose 10-20% overhead due to VM, who cares ?

Re:Virtualization is good enough

[Mr. Flibble]

I have done consulting for a number of $LARGE_BANKS and seen exactly what you describe. I am dealing with one large company now that has 3 servers allocated to run a SINGLE piece of software. They don't need 3 servers to run it, and I suggested that they incorporate these 3 machines into their ESX network, but like you just mentioned, they want "failover" and "enough resources". Never mind that I have the same software running now on a single lower server that is running ESXi 3.5 with 9 other VMs on the same machine.

Re:Virtualization is good enough

[Kjella]

It's CYA in practise. Here's the usual chain of events:

1. Business makes requirements to vendor: We want X capacity/response time/whatever
2. Vendor to business side: Well, what will you do with it?
3. Business makes requirements to vendor: Maybe A, maybe B with maybe N or N^2 users
4. Vendor to business side: That was a lot of maybes. But with $CONFIG you'll be sure

Particularly if the required hardware upgrades aren't part of the negotiations with the vendor, then it's almost a certainty.

Re:Virtualization is good enough

[Anonymous Coward]

This isn't an issue of 10-20% overhead, this is an issue of the application running ten times slower (RTFA). If you think a VM only adds 10-20% overhead, you are deceiving yourself and have not actually measured. It's like the difference between a compiled and an interpreted language all over again.

Re:Different Operating Systems

[Anonymous Coward]

In your scenario, I'd recommend running DBAN

Well, duh!

[www.sorehands.com]

You ask that the OS be put into a virtual machine, would you not expect a big performance hit??? It is only common sense to anyone with any basic computer knowledge. You are adding another layer between the hardware and the program, what do you think would happen?

Re:Well, duh!

[Gothmolly]

A real hypervisor like used by IBM on their p-series frames doesn't impose this penalty. You're thinking of an emulator.

Re:Well, duh!

[Anonymous Coward]

Not necessarily. No expert here, but the means by which you are virtualizing has an effect. The hardware on which you are virtualizing makes a tremendous difference. Visit Sun's site, and pull up everything on VirtualBox. There is a downloadable PDF - "Virtualization for dummies" which I read through last night. Other documents are available, just browse around, and grab them to read. Feel free to search VMWare's site for similar documents, but read.

Yes, almost all VM's today take a performance hit. But, I have two VM's running on my desktop right now at the same time. I still have 20 % real physical memory available, and the CPU jumps from 60% to 80%, depending on what I'm actually doing.

The machine is working pretty closer to capacity than it ever does with only the host machine running, and performance is "good" on all three. Not "excellent", but "good". I won't tolerate thrashing to virtual memory, so the trick is to have enough memory.

Adding one more VM would almost certainly overload my system, causing continous thrashing, and I would simply give up by closing one of them.

On a server, you don't go cheap on memory - you load the thing up. It makes sense to virtualize a machine that sees little traffic, rather than buying all new hardware for it.

With VMWare infra, scripts can keep up with memory and CPU utilization, and actually start up an additional physical machine for the purpose of offloading one or more VM's when the load gets heavy.

As CPU's continue to be developed, and as the software evolves, you can expect virtualization to make more and more sense.

Re:Well, duh!

[JustNiz]

Uhh I call bullshit. There's no such thing as a hypervisor that has absolutely no performance penalty over running native.

Re:Well, duh!

[chez69]

Everything has some overhead, sure. However, when you have hardware level virtualization (where the logic is in the firmware like the IBM mainframe systems), and not in some software hypervisor, the overhead is very minimal. On mainframes they where running LPARs with native performance over 20 years ago.

Re:Well, duh!

[Alrescha]

"There's no such thing as a hypervisor that has absolutely no performance penalty over running native."

Actually, back in the day, it was sometimes faster to run an operating system under VM (like DOS/VS) as the hypervisor was smarter about resource management than the guest OS was.

A.

It's the Apps more than the OS!

[Anonymous Coward]

Disclaimer: TFA was down (slashdotted) so I didn't bother to read it.

You're thinking too much about the OS and not enough about the apps, which are the entire reason why we have computers.

If applications were written well, and played nice with others, and had realistic sizing requirements/guidelines then I could see your point. However a lot of apps frankly are poorly written with this idea of 'I can do anything on my OS that I want to.' That leads to having to silo applications as well as oversized servers (just throw lots of hardware at my inefficient program).

Not to mention that in general there are certain workloads that are more appropriate for certain varying OSes, which can also vary depending on the IT staff supporting said applications. I don't want to have separate hardware for each different OS.

The underlying OS architecture can matter somewhat, but until developers write better apps it's not as big a deal as one might think would be my 2 cents.

Crappy programing beats virtualization overhead cost. (though I'd love if that wasn't the case!)

I've seen this before

[bertok]

I've seen similar hideous slowdowns on ESX before for database workloads, and it's not VMware's fault.

This kind of slowdown is almost always because of badly written chatty applications that use the database one-row-at-a-time, instead of simply executing a query.

I once benchmarked a Microsoft reporting tool on bare metal compared to ESX, and it ran 3x slower on ESX. The fault was that it was reading a 10M row database one row at a time, and performing a table join in the client VB code instead of the server. I tried running the exact same query as a pure T-SQL join, and it was something like 1000x faster - except now the ESX box was only 5% slower instead of 3x slower.

The issue is that ESX has a small overhead to switching between VMs, and also a small overhead for estabilishing a TCP connection. The throughput is good, but it does add a few hundred microseconds of latency, all up. You get similar latency if your physical servers are in a datacenter environment and are seperated by a couple of a switches or a firewall. If you can't handle sub-millisecond latencies, it's time to revisit your application architecture!

Re:I've seen this before

[stefanlasiewski]

He is not using ESX. He was using VMware Server, running on Ubuntu. I would expect that to be slower then VMware ESX or FreeBSD Jails.

Not saying anything bad about BSD Jails....

[Anonymous Coward]

"we added more database indexes..."

I have no experience with BSD Jails so I can't comment, but...

YES IF YOU ADD NEEDED INDEXES TO A SIGNIFICANTLY SIZED DATABASE A 10X PERFORMANCE INCREASE (OR EVEN FAR GREATER) IS NOT UNHEARD OF

Re:Not saying anything bad about BSD Jails....

[BitZtream]

They added indexes and it didn't matter. You're reading comprehension ability is absolutely asstastic. They got 10x the performance when they moved from VMs to real hardware and jails.

Which however results in the same sort of response from me:

No shit? You got way better performance when you removed a useless layer of overhead? Never would have figured that one out.

I am not saying that virtualization is evil.

[www.sorehands.com]

As with everything, there is always a trade-off. I run OS/2 on my laptop inside of a virtual machine. The reason I do that is to be able to run Windows apps and not have to deal with some of the lack of OS/2 device drivers for some hardware.

Re:I am not saying that virtualization is evil.

[JustNiz]

You describe why you run OS/2 in a vm, but the real question your comment poses is why you or even anyone would want to run OS/2 at all these days.

It's in the bios and the chip choice

[Nefarious Wheel]

The difference between a good ratio and a bad ratio between vm's and hardware is often due to the need in some combinations to run every vm's IO state through the BIOS in order to complete an IO. That's a lot of interrupt state passing and subsequent process rescheduling. You get a multiplier when multiple vm's are all competing for the same trap completion and queues grow as a result. I know that Intel at least has a chip set that optimises this (they call the feature VT) . Ring 0 instruction completion has a huge multiplying effect on virtualisation efficiency. Right chip set = good, wrong chip set = sorta.

Wrong tool for the job

[FranTaylor]

You might as well have said,

"Our earth moving business took a big jump in productivity when we switched from ice-cream scoops to backhoes".

Virtualization is a gift for Windows servers!

[JakFrost]

I've worked for many of the Fortune 10 (DB, GS, CS, JP, MS, etc.) banks on the Windows server side and they are all going full steam ahead for virtualization with VMWare or Xen exactly because they have been buying way too much hardware for their backend applications for the last decade. The utilization on all of these servers hardly hits 5-10% and the vast majority of time these systems sit idle. The standard has always been rackmount servers with multiple processor/core systems with gigs of memory all sitting around being unused, mostly Compaq/HP systems with IBM xSeries servers and some Dells thrown in for good measure.

The reason that this over-capitization has been the requirement of the business line departments to choose only from four or five server models for their backend application. These standard configs are usually configured in rackmount spaces 1U, 2U, 3U, and 4U sizes and with nearly maxed out specs for each size and the size of the server determines the performance you get. You have a light web server you get a blade or a pizza box, you have a light backend application you get a 2U server with two processors or four cores even though you might have a single threaded app that was ported from MS-DOS a few years ago, you want something beefier you get the 4U server with 4 processors, 8 cores and 16 GB of RAM even though your application only runs two threads and allocates 512MB of ram maximum. I've monitored thousands of these servers through IBM Director, InsightMangager, and NetIQ for performance and 99% of the time these servers are at 2% processor and memory utilization and only once in a while for a short amount of time one or two of the cores get hit with a low-mid work load for processing and then go back to doing nothing. These were the Production servers.

Now consider the Development servers, where a bank has 500 servers dedicated for developer usage with the same specs as the production boxes and at any one time maybe a few of those servers get used for testing while the other few hundred sit around doing nothing while the developers get a new release ready for weeks at a time. The first systems to get virtualized were the development servers because they were so underutilized that it was unthinkable.

(Off topic: Funny and sad story from my days in 2007 at a top bank (CS) helping with VMWare virtualzation onto HP Blades and 3Par SAN storage for ~500 development servers. The 3Par hardware and firmware was in such a shitty state that it crashed the entire SAN frame multiple times crashing hundreds of development servers at the same time during heavy I/O load. The 3Par would play the blame game against other vendors accusing Brocade for faulty SAN fibre switches, Emulex for faulty hardware and drivers, HP Blade and IBM Blade for faulty server, and the Windows admins for incompetence. Only to find that it was their SAN interface firmware causing the crashes.)

VMWare solves the problem of running commercial backend applications on Windows servers since each application is so specific due to the requirements of the OS version, service pack, hotfixes, patches, configurations that the standard is always one-server to one-application and nobody every wanted to mix them because any issue would always be blamed on the other vendor's application on the server. There were always talks from management about providing capacity to businesses that is scalable instead of providing them with single servers with a single OS. That was five years ago and people wanted to use Windows Capacity Management features but they were a joke since they were based on per-process usage quotas and the of course nobody wanted to mix two different apps on the same box so those talks went nowhere.

That is until VMWare showed up and showed a real way to isolate each OS instance from another while it also allowed us to configure capacity requirements on each instance while letting us package all those shitty single threaded backend applications each running on a separate server onto on

Re:Virtualization is a gift for Windows servers!

[drsmithy]

(Disclaimer: This is written by a Windows server admin who wished he was a Unix server admin instead.)

The situation is not much different. We're virtualising everything for the same reasons.

Linux-Vserver

[Daniel15]

For something similar for Linux, take a look at Linux-Vserver. I've been using it for a while, it's pretty good. A while ago, I wrote a howto showing how to install Linux-Vserver on Debian Etch, most of it would still apply today :)

microkernel + ACL

[bzipitidoo]

Chroot and jails? Hypervisors? Sounds like what we really need is a microkernel OS with decent security features, like Access Control Lists.

BSD is not dead

[commodoresloat]

It's just in jail.

Re:BSD is not dead

[Chrisq]

That's nice. It will give Hans Reiser something to do.

Re:BSD is not dead

[ArsenneLupin]

That's nice. It will give Hans Reiser something to do.

Unfortunately, Hans Reiser prefers chicks...

There can be huge differences in performance

[nickh01uk]

Theres a nice little article here (basic reg. required) contrasting VMware and Citrix XenServer, where the end user was forced to abandon VMware (their default choice) after suffering performance problems and after 6 months of back and forth with tech support and engineering at the vendor. In the end XenServer delivered 2x the real world performance on identical hardware with a default install. Not all workloads are equally well virtualized! N.

ICore Virtual Accounts

[Ostracus]

iCore Virtual Accounts Container based virtualization for Windows.

Re:ICore Virtual Accounts

[Psyko]

Has anyone actually played around with this? This looks like it has potential for some sandbox stuff I've been thinking about on workstations.

NFS

[kasperd]

We had heard before that I/O performance and disk performance are the weaknesses of virtualization but we thought we could work around that by putting the job databases on an NFS export from a non virtualized server.

Sounds to me like they heard about some potential performance problem, and without understanding that problem or trying to compare performance of various solutions, they decided NFS was the solution for that performance problem.

Did they ever try using the virtual block devices provided by the virtualization rather than the NFS solution? My guess is that NFS was actually the reason for their performance problems.

Re:Different Operating Systems

[DavidRawling]

I've run both on comparable hardware. Hyper-V was way, way better, performance-wise if you had >3 or 4 running VMs simply because it didn't have the extra scheduling overhead. In either case though, if your disks are slow the VMs will be too.

SQLite? Huh?

[Jacques Chester]

As much as I like and admire SQLite, I'm not sure if it's the right tool for the job. Something like PostgreSQL, with proper MVCC and nice multicore scaling, would probably have worked a lot better in the first place.

OpenVZ

[billysara]

OpenVZ is often overlooked for this kind of workload. _Kind_ of similar to a jail environment. We use it for a lot of "light" servers - project websites, that kind of thing but it will handle a lot more than that. http://wiki.openvz.org/Main_Page . Easy to install, really easy to configure & manage.

Re:Different Operating Systems

[Gumbercules!!]

Thanks very much for your reply. :-) I will give a go to Hyper-V and see how it works out, as I plan to run 5 concurrent VMs.

Disks should be fine - they're just desktop grade 7,200 SATAs but I don't expect the VMs to do much more than idle for 90% of the day.

Oh my God

[Slashcrap]

They actually used VMware Server 1 for a production site. No, seriously they really did. That's what's being compared here. Is FreeBSD even a supported guest?

FreeBSD users are either a well organised trolling group or ridiculously bad at advocacy. Surely this was designed to cause outrage amongst those who know what they're doing? I just can't believe it's accidental.

How does your OS do when it isn't racing a crippled child?

Terrible name

[dugeen]

'Jail' is such a terrible metaphor to choose for a product. I want a happy metaphor like 'sandbox', not something redolent of brutality, despair and iron sorrows.

Re:Terrible name

[jellybear]

I agree. How bout "Big House"? That would make me think of a house that's big.

Re:Terrible name

[Just Some Guy]

'Jail' is such a terrible metaphor to choose for a product. I want a happy metaphor like 'sandbox', not something redolent of brutality, despair and iron sorrows.

I want our applications to be too freaking terrified to even consider trying to escape.

Linux alternatives

[speedtux]

There is no reason to switch to BSD just to get this functionality; Linux has plenty of choices for isolating software, allowing all sorts of tradeoffs between performance and isolation.

If you want something more lightweight than VMware/VirtualBox, you have plenty of choices on Linux: KVM, AppArmor, vserver, OpenVZ, LSM, SELinux, or even the BSD jails patch.

VMWARE server is bad choice for this app

[Anonymous Coward]

Why would anyone use VMware Server in a production environment? It is not meant to be a heavy-duty hypervisor as it runs on top of another OS.

ESX 3.5 and 4.0 are meant to run mission critical applications.

Anandtech ran up 8 VMs on top several server CPUs, including the new six-core. Those 8 VMs include a heavy Oracle OLTP database (2x) and 100 GB large SQL Server database:
ESX 3.5 and 4.0 benchmarks

  VMware ESX is about running several instances of Windows and Linux on top of the same machine and manage them easily. FreeBSD Jails are of course a better solution if you just want lots and lots of small machines and the management overhead is zero. Jails are what is called "Container based virtualization".
Container based virtualization

MTA - GUI? Huh?

[Anonymous Coward]

Why would an MTA have a GUI? That seems foolish to me.
There may be a need to have a GUI for the config files, but many would just use vi.

ESXi hasn't been free even for a year. I was at the announcement and had to wait about a week before it became available. That was last July/August, I believe. ESXi wouldn't load on any of my servers. It didn't recognize our disk controllers. Seem if you don't use HP/Dell/IBM servers, you definitely need a SAN. Don't get me wrong, iSCSI Rocks, but we didn't have a SAN at the time.

DBAs and Virtualization

[Bigmilt8]

You wasted your time. I'm a DBA with a programming background. Virtualization is not suitable for mid- to large- database environments. Database software is designed to handle all IO and memory issues internally. The virtualization software just gets in the way.

Re:DBAs and Virtualization

[LWATCDR]

Virtualization is not suitable for mid- to large- database environments.
He is running SQLLite.

Server is a bad choice for a production server!

[DecepticonEazyE]

They compared it to Server v1? That's an unfair comparison. When you stack a hypervisor on top of another OS, yeah, there's going to be overhead. Probably too much overhead for a production server. Compare it to ESX3.5 or even ESX3i, then we'll talk.

Re:Silly faggots

[Anonymous Coward]

You forgot option 3) Card carrying member of the KKK. Most likely the Grand Dragon.

Re:Different Operating Systems

[DavidRawling]

You might be surprised. I have had 6 VMs running on a Hyper-V server with 8GB of RAM; the disks were 500G SATA 7200rpm on the Intel ICH9R RAID controller, RAID 10 (4 disks). Performance is great, even when snapshotting and so forth, or running 8 VMs at full memory commit.

I have another Hyper-V box with 8GB of RAM and the nVidia RAID controller; running 2 VMs on a RAID 1 set was atrocious.

Re:Different Operating Systems

[Gumbercules!!]

Ahh. More useful info thanks. I would have been using an nVidia (fake) RAID controller on RAID-1, too.

I think I will give it a go and see what happens but thanks for your useful info.

I just wish I hadn't started my initial comment with a silly joke that got me modded redundant so that more people might have seen my comment and provided useful feedback, like you did. :-P

Solaris

[JAlexoi]

In their place I would checkout the Solaris equivalent of jails. Solaris Zones look really good.

It depends on the amount of compartmentalization.

[argent]

Jails aren't the low end, even. UNIX is a multiuser environment, and simply running multiple instances of the server in separate directory trees provides all the isolation you need. If that's not enough, you can use chroot, then jails or the various equivalents on Linux, then lightweight VMs, and full VMs, blades, separate servers...

It's a continuum. The best solution depends on the overhead you can afford to lose and how tight the compartmentalization has to be. For a lot of problems I've seen people using VMs to solve, even jails are kind of heavyweight.

Windows, of course, has a different application model and it's harder to use some of these intermediate solutions... but you should still be able to do things like run multiple Apache servers bound to different addresses in different directory trees and user IDs, instead of taking on the overhead of a VM.

Let's try that again...

[argent]

I wrote: "simply running multiple instances of the server in separate directory trees provides all the isolation you need".

Of course, I meant: "simply running multiple instances of the server in separate directory trees often provides all the isolation you need".

Note to self: preview your posts, idiot.

Wait, What?

[ConallB]

They were running thier outfit on VMWare Server - as in the free-runs-on/in-a-desktop version. So basically thay benchmarked a virtual machine runnning on top of a full blown OS running on physical hardware. Then they switched to running a virtualised environment on physical hardware. I am amazed that they ony got a 10 fold increase! Seriously, Try ESXi, the free hypervisor from VMWare and run the benchmarks again. The author really should do more research before slating vmware.

Don't run a database in a VM

[Evro]

I learned this myself, and just don't do it. VMWare is awesome for CPU- or memory-intensive tasks, but for IO-intensive stuff like databases it's horrible. At least that's my experience, with ESX 3.5 and an iSCSI SAN.

Re:Don't run a database in a VM

[Mistoffeles]

Have you eliminated every other possible factor? After all, if your database is on a SAN, it is completely unaffected by the VM running on the machine that accesses the SAN.

It could be that you need to optimize either your SAN configuration or your database access configuration, or even possibly be your indexes (this being said not knowing your experience level in database management and index optimization).

this just in

[mistahkurtz]

some things may not be a good candidate for virtualization. ESPECIALLY in a virtualization client that loads ON TOP OF AN OS.

i'm wondering where they got their performance expectations from. FTFA, they didn't buy the vmware tools that would give them the performance and capabilities they desired.

so, to sum it up, they virtualized an application or applications they possibly shouldn't have. they didn't pay for the set of tools that would have given them the best shot of acheiving their virtualization goals, and instead used what was quite possibly the worst tool for the job. now they're complaining about it, and pointing out how (apparently) just running a single OS on the servers, with multiple application instances.

what a waste of time. this is not news. don't use wireload or yippiemove. they, according to their own words in TFA, don't have a clue what they're doing.

VMWare Server 1.x is a dog

[shogarth]

Ever since the launch over a year ago we used VMware Server 1 for instantiating the YippieMove back-end software.

This says it all in one sentence. VMWare Server (as opposed to ESX or ESXi) is a dog. It barely ran with two WinXP installs and one RHEL5 on a 4-core server with 8 GB of RAM. Life was a little better after upgrading to VMWare Server 2, but running it on top of an OS instead of using a hypervisor kills performance. I switched the same box over to ESXi 3.5 and all three installs scream. Additionally, the memory page deduplication driver means that I have capacity for probably another five to seven lightly loaded systems without worrying about the occasional load spike.

As far as some jobs just not being well suited to virtualization, that's an obvious truth. However, most work in that class is CPU bound compute work. If you are not buying storage on a shoestring budget (i.e. you can run VMFS3 on a FC or trunked Gb SAN rather than fiddle with NFS) then you should have reasonable IO performance. The OP doesn't give any detail on storage performance (either in bandwidth or IOPS) so there's no way to tell what it requires. Having looked at his YippieMove service web page there doesn't seem to be a lot that is required. It seems like they picked the low performance, free VMWare tool and when it didn't work did something completely different. This says less about VMWare than it does about the OP's design/testing process.

"FreeBSD really isn't intended for the desktop PC"

[devloop]

While I agree with you on FreeBSD being a fine (if not *the* finest) server OS,
it really irks me to see the "not intended for desktop use" myth being perpetuated.

The desktop *is* one of the explict targets for FreeBSD:

        "With over 20,000 ported libraries and applications, FreeBSD supports applications for desktop, server, appliance...."
                http://www.freebsd.org/about.html

      "*BSD makes a great server. It also makes a great desktop....
        *BSD has access to the same desktop tools (KDE, GNOME, Firefox, windowmanagers) as Linux.
          And ``office'' applications such as OpenOffice suite work under *BSD too."
                http://www.freebsd.org/advocacy/myths.html#server

FreeBSD 7 in fact, among other things, had *major* wireless rework done,
a feature almost exclusive to desktop use.

The aggressive demeanor of your post is shameful and in contrast with
the overall spirit of the *BSD community.

To which I would add...

[Anonymous Coward]

IBM has, after all, been working on virtualisation for nearly 40 years. (for the GP's benefit)

howto: Rev-UP VMWare Server/Wkstn in Linux Host OS

[IBitOBear]

Okay, I have been through this at work several times recently. There are two major slow-downs in the default (but reasonably bullet-proof) VMWare machines running on a Linux _host_.

1) If you are doing NAT attachment _don't_ use the vmware NAT daemon. It pulls each packet into userspace before deciding how to forward/nat it. So don't use the default nat device (e.g. vmnet8). Add a new custom made "host only" adapter (e.g. vmnet9-or-more) by adding another adapter, and then use regular firewalling (ip_forward = 1 and iptables rules) so that the packets just pass through the Linux kernel and netfilters once. (you can use vmnet1 in a pinch but blarg! 8-)

1a) If you want/need to use the default nat engine (e.g. vmnet8) then put the nat daemon into a real-time scheduling group with "chrt --rr --pid 22 $(pgrep vmnet-natd)". Not quite a good as staying in the kernel all the way to your physical media.

1b) if you do item one, don't use the vmware-dhcpd, configure your regular dhcpd/dhcpd3 etc daemon because it will more easily integrate with your system as a whole.

(in other words, vmware-dhcpd is not magic, and vmware-natd is _super_ expensive)

2) VMWare makes a /path/to/your/machine/machine_name.vmem file, which is a memory mapped file that represents the RAM in the guest. This is like having the whole vm living forever in your swap space. It's great there if you want to take a lot of snapshots and want to be more restart/crash/sleep safe. It _sucks_ for performance. If you use "mainmem.usenamedfile=FALSE" in your .vmx files. (you have to edit the files by hand). This will move the .vmem file into your temp directory and unlink it so it's anonymous and self-deleting. It slows down snapshots but...

2a) Make _SURE_ your /tmp file system is a mounted tmpfs with a size=whatever mount option that will let the tmpfs grow to at least 10% larger than the (sum of the) memory size of (all of the) vritual machine(s) you are going to run at once. This will cause the "backing" of the virtual machine RAM to be actual RAM and you will get rational machine RAM speed.

2b) If you want/need to, there is a tmpDirectory=/wherever diretive to say where those files go. It gangswith the usenamedfile=FLASE and you can set up dedicated tmpfs files to back the machines specially/separately.

2c) If you want/need the backing or have a "better" drive you want to use with real backing, you can use the above in variations to move this performance limiter onto different spindle than your .vmd (virtual disk files).

3) No matter what, your virtual memory file counts against your overcommit_ratio (/proc/sys/vm/overcommit_ratio) compared to your ram. It defaults at 50% for _all_ the accounted facilities system-wide. If you have 4Gig RAM and try to run a 3G vm while leaving your overcommit_ratio at 50, you will suffer some unintended consequences in terms of paging/swapping pressure. Ajust your ratio to like 75 or 80 percent if your total VM memory size is 60 to 65 percent of real ram. _DONT_ set this nubmer to more than 85% unless you have experimented with the system stability at higher numbers. It can be _quite_ surprising.

Anyway, that's the three things (in many parts) you need to know to make VMWare work its best on your linux host OS. It doesn't matter what the Guest OS is, always consider the above.

Disclaimer: I don't work for VMWare etc, this is all practical knowledge and trial-n-error gained knowledge. I offer no warranty, but it will work...

Er, nope its the way vmWare does RAM in Linux Host

[IBitOBear]

I covered this in more detail in a top level post below. The actual performance problems for most vmWare machines (typically) is not the virtual disk. The way vm machine memory is backed by a mapped regular file, combined with the way large-memory VMs interact with the overcommit_ratio on a Linux host OS (e.g. running vmware under linux, regardless of what the vm is running internally) produce almost all of the slowdowns.

The way VMWare does NAT through a userspace daemon isn't the best thing on the planet either.

(read the how-to post below for the remedies instead of looking here for re-pasted text, I'm not _that_ much of a karma whore 8-)

Extra pointer(s) for Windows Guests

[IBitOBear]

If you are running a windows guest, regardless of whether you are running a Linux or Windows host you should:

1) _Definitely_ leave paging turned on in the windows guest.

2) If you have multiple physical disk devices, make a "D:" drive as a separate virtual disk where the virtual disk image files are on a different physical spindle in the host os. Now inside the windows guest delete the page file from C: and create one on the new/separate D: drive (who's files are on the "the other spindle" in the host os).

3) If you _NEVER_ want to "suspend" the windows guest (you can still suspend the host separately) you can do the following advanced trick...: [NOTE: this may be worth doing even if you don't have a separate spindle for D:]

3a) Create a virtual disk to mount as D:
3b) Start windows guest OS.
3c) Log in as an administrator and Format the D: drive (with all that entails); FAT is fine and possibly optimal as the file system type.
3d) Go into the windows VM settings and delete the windows pagefile from C and make a large fixed or system-managed page file on D:
3e) Windows will tell you that you have to reboot for this to take effect, but _DONT_, shut down windows instead.
3f) Go into the settings for the virtual disk that you just made the D: drive and make it "non-persistent".

You now have a eternally empty "D:" drive, its only contents is the freshly formatted disk image. Every time windows starts it will know that there _should_ be a page file on D: and it will create that page file anew on an "empty" drive, so it will always be defragmented etc and you can move the windows vm around more easily (like it will zip/tar up smaller) since the random contents of the pagefile will always be discarded on shut down. D: is also now the _ultimate_ self cleaning /tmp directory for windows. If you move your TEMP and TMP environment variables to point to D:\ (or if you make other directories on D: before step 3e,) they will always be empty on reboot, including whatever cache files you put there etc.

As an aside, all windows instances everywhere should have "pagedefrag" installed on them (search microsoft.com for this application) whether they be Host or Guest OS instances.

Also, not to plug yet-another product I have no interest in besides finding it useful, jkDefrag is the best and works particularly well after pagedefrag and before a vmware shrink operation.

BSD is alive?

[Anonymous Coward]

So BSD is still alive huh?

Re:Government IT is being poisoned by virtualizati

[Anonymous Coward]

The new buzzword of Virtualization has reached all corners of the US Government IT realm. Blinded by the marketing hype of "consolidation" and "power savings" agencies of the three-letter variety are falling over themselves awarding contracts to "virtualize" the infrastucture. Cross-domain security be damned, VMWare and Microsoft SoftGrid Hyper-v Softricity Whatevers will solve all their problems and help us go green at the very same time, for every application, in every environment, for no reason.

Both zones on Solaris 10 (+Trusted Extensions) and VMware are rating MLS-capable and can be used for cross-domain security. Solaris can also do CIPSO tagging on the network as well AFAIK.

If you want to take it for a spin, Solaris 10 (and TX) run under VMware just fine, and so you can play with MLS on your own system.