Slashdot Mirror
The Trouble with Virtualization - Cranky IT Staffs
lgmac writes "A new survey on the results of Enterprise use of virtualization shows that the process is seeing wide and appreciative use. Technical hurdles are obviously the biggest problem facing corporate IT shops. Just the same, political squabbles among IT staffers fighting for turf after being forced to work together in new ways seems to be a going concern as well. 'Technical woes rank higher--to be expected when CIOs deploy a new technology such as virtualization. However, the politics pain many of you. Remember, virtualization not only asks people to cede some control over their physical server kingdoms, but also asks IT experts from different realms to work more closely together.'"
34% of surveyed companies have been running virtualized desktops? Putting aside that that number doesn't seem to square with the "Virtual Desktops a Hard Sell" table below, does that seem likely?!?
What I'm listening to now on Pandora...
My companies biggest problem concerning virtualization at this point has been backing up running copies of virtual server without interruption, anyone have some insight on this?
Technology is continually changing. Those who adapt will be the most successful. Those who don't will eventually be pushed aside. Fighting over turf won't get you far in a corporate environment in the long term.
Developers: We can use your help.
In my experience as a systems engineer, the biggest problem we've had with virtualization is that too many people who don't understand it well view it as a magic wand that you can wave to make all your capacity & provisioning problems disappear.
"Hey! We need a new server to run Blah version 3.0!"
"No problem! Sammy can create a new virtual server!"
"Oh wait - my bad. We actually need a whole farm."
"That's okay, he can whip up a whole batch of them!"
Ad nauseaum. About the worst I've heard was a clueless manager asking me if the resource requirements for Oracle 10g could be relaxed because we were running it on VMware. I actually found myself calling a "come to Jesus" meeting in which I explained, in as simple terms as I could, that "making the system virtual" doesn't mean that hardware requirements go away. Very, very few applications get faster when you put them on equivalent hardware, only virtualized.
My company works with several shops that are working on large-scale virtualization and common platform projects. I would say the biggest single issue is simply politics, because much of the initial work is affecting older platforms that are the biggest win technically and financially to replace. For instance, one shop has a significant investment in Alpha systems, and still has production servers that are 15+ years old running a huge chunk of their revenue producing systems. The folks working directly on the Alpha servers have considerable clout, since they've been the golden children for many, many years. Their bosses know how to play politics, and, considering that Alpha/VMS experience is one of those IT areas where there is little new blood from younger IT staff members, they are quite adept at finding reasons why it won't work to serve their own ends.
Not only that, but virtualization will result in lost jobs at some point. Many IT staffers are afraid, whether rightly or wrongly, of losing their jobs. In a sense, they are outsourcing a good chunk of their day-to-day duties. I remember when this particular company went to SAN's over the last half-decade, and you would have thought, from the way the Alpha guys were fighting it, that the world was ending. They created road-block after road-block about how they wouldn't be able to keep the systems running, how it wouldn't work in "their" environment, etc, etc.
And, because of the compartmentalization that often occurs in large enterprise, many of these guys have very little idea about anything outside their own box. I know guys who have architected corporate platform migrations who are so narrow in their focus that they have *NO* experience outside their box, be it a particular OS, a server type, a network type, whatever. When the box becomes a cloud of equipment, they are lost and often have little or no ability to work with the other layers involved. Learning new troubleshooting skills in these environments is a painstaking process, and not one that many people are comfortable with.
In the end, these various factors are creating far larger artificial roadblocks for implementing virtualization than any technical challenges. To top it off, much of this is being driven by financials. The CFO and CTO are desparately trying to find ways to cut costs. By the time this message percolates down to the workers, they feel threatened rather than empowered, and have little incentive (and generally no training, either) to be complicit in what they feel is a threat.
Bill
It may be more practical to back up the system from within the VM, i.e. treat it as if it weren't a VM. By definition this will be on a live system.
Another option:
Have your VM use a checkpoint disk. Once a day shut down the VM, merge the changes from this week into the checkpoint disk, and restart the VM. This may take anywhere from a few minutes to tens of minutes. Restart the VM. Back up the checkpoint-disk image.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
This is a problem with management and/or the IT staff.
Management should run the company in a way that cooperation is rewarded not punished. Consolidation to save money shouldn't result in harm to those who are making it happen or anyone else for that matter.
The IT staff as well as all of the other employees and officers should have the attitude that if it's good for the company and not bad for anyone else it's the right thing to do.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
VMware has multiple ways to balance and protect resources. You can set hard limits on VM resource utilization, ensuring that one machine can never take over a certain percentage of CPU, memory and even network bandwidth. VMs can also be given "shares", which determine priority over resources. In a contention for resources, the VM with the highest number shares is given immediate access to what it needs, with the lower share VMs splitting what's left over. This is the recommended way to handle it, as it gives you the best overall hardware utilization across your entire implementation.
Starting in VI3, VMware also introduced the ability for VMs to migrate automatically across an entire farm of hosts, based on server load. In my experience, with very little tweaking, VMware does a very good job of fairly balancing resources.
Items that need to be redundant, should not be virtualized on shared hardware. I've heard people want to virtualize redundant instances of directory services, databases, proxy servers...etc. I call this the "putting all your eggs in one,central-point-of-failure, hardware basket".
If you're doing something stupid like putting clusters or redundant servers on the same virtualization host, then I would agree. High availability loses it's meaning if all your nodes have a single point of failure.
However, there's absolutely no reason you can't make your virtualization implementation highly available itself. Right now, I have clusters running in VMware VI3, that are running on separate hosts. Even with DRS, which balances all your VMs across an entire pool of servers, I can ensure that redundant servers and clusters don't end up running on the same piece of physical hardware. And when you add HA into the mix, you also provide a level of high availability to systems that you might not otherwise have been able to justify the expense on.
An effect I've noticed many times is that when you ask IT staff to vote, the windows IT staff always outnumber the Unix and mac It staff. Thus one man one vote favors the windows firedrill fix-it jockeys over the more talented kernel of Unix and mac support gurus. Yes I realize that's ripe for flamebait, but it's actually true. By and large windows has so many problems to keep functioning it lakes a large staff of low paid trained monkies on hand. The revenge of the c-strudents is that they out number the A-students who run the linux servers.
You want to watch a fight? Get the Windows Server sysadmins and the UNIX/LINUX sysadmins and ask each group which server OS should be the "Native" operating system under which the other runs....fun...
Nobody should EVER virtualize a server that needs special hardware. EVER. Virtual servers should be reserved for the most generic of hardware requirements. Once you start bringing in fax boards you need a dedicated physical solution. If you want to test that kind of thing, go ahead and virtualize it, but the production box should be physical. I shudder to think of a virtualized firewall or router. Ouch.
I don't know about you, but my servers run on the power of cotton candy and happy thoughts. -Anonymous Coward
This is a classic sign of a broken IT department. One place I worked, if you (well, if I) needed to increase the size of a database table, I had to get sign-offs from
net result? nothing ever got agreed. The simplest changes took forever and cost a fortune. The operation is now outsourced.
Who's to blame? Probably not the techies, they just pressed buttons. Quite likely the team-leaders for turning it political, definitely the IT managers who allowed the situation to continue.
Who kept their jobs?
yup, the managers! You've been warned: infighting only hurts the foot-soldiers, the generals aren't affected. Sort it out yourselves or you'll have to start learning chinese.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
But the IBM commercial told me that i could replace my entire datacenter with a single server! Are you telling me that those two youngish looking racially diverse guys having a conversation at a coffee shop about sloshing hundreds of thousands of dollars worth of hardware around even though neither of them really seems to be sure what all of it does were LYING TO ME!?
NewslilySocial News. No lolcats allowed.
Say you have a need to add another fax board(or whatever) to the virtualized x86 server, to find that they stuck some mission critical Virtual Environment on the Server and It CAN'T come down for another 2 weeks.
:) If you need to do work on a physical server [say hardware replacement, firmware upgrade, etc], you put that server into maintenance mode. VMWare will automatically and **transparently** migrate the running VMs onto the remaining servers. You can then power off the physical server, do whatever you need to do, and power it back on. When it comes back up, take it out of maintenance mode, and VMWare will automatically start migrating VMs back onto it to balance the load.
Aside from another poster's excellent point about not virtualizing servers that require specialized hardware, you're missing another point of the virtualized servers.
In the case of VMWare ESX server, you'd use VMotion to solve this problem. Say you have a cluster of 3 or 4 physical servers running some number of VMs. Hopefully you're not dumb enough to have all those servers running anywhere near 100%.
Nobody will notice the fact that their mission critical server just moved from one box to another. Worst case, if your servers were already a bit too heavily loaded, some applications will slow down a bit while you're doing maintenance, but a temporary slow down is a lot better than having an application completely down every time you have to upgrade firmware or replace a stick of RAM.
As an added bonus, if you so choose, VMotion can automatically balance the VMs at all times, so if one particular VM is suddenly requiring a large amount of resources, VMotion can migrate it to a less heavily loaded box, or migrate other less needy VMs to another box to free resources for the VM that needs it. This is great for handling short-term usage fluctuations, or can even be scripted to adjust for known, regular usage peaks.
That's not to say there aren't downsides to virtualization, but the situation you described isn't one of them.
Nothing to see here
Many large virtualized deployments include very advanced technologies such as shared SANs, shared infrastructure, and complex virtualization tools.
Frankly, I would argue that you are probably just redeploying people resources into different roles and responsibilities, while probably saving on hardware and energy costs for the infrastructure through consolidation.
Lindsay Blanton
RadioReference.com
We recently moved everything into virtua-land, complete with a hige SAN, fiberchannel switches, blade servers - the whole nine yards.
While I do think the move was a net positive, the complication of 60 physical servers was more or less replaced by the complication of all the new SAN/Bladecenter components and their interdependency.
One particular thing we've run into is "firmware hell", where you have several components in the chain that all require firmware updates and all depend on each other.
I don't always use unix-like operating systems; but when I do, I prefer FreeBSD.