Slashdot Mirror
The Trouble with Virtualization - Cranky IT Staffs
lgmac writes "A new survey on the results of Enterprise use of virtualization shows that the process is seeing wide and appreciative use. Technical hurdles are obviously the biggest problem facing corporate IT shops. Just the same, political squabbles among IT staffers fighting for turf after being forced to work together in new ways seems to be a going concern as well. 'Technical woes rank higher--to be expected when CIOs deploy a new technology such as virtualization. However, the politics pain many of you. Remember, virtualization not only asks people to cede some control over their physical server kingdoms, but also asks IT experts from different realms to work more closely together.'"
Technology is continually changing. Those who adapt will be the most successful. Those who don't will eventually be pushed aside. Fighting over turf won't get you far in a corporate environment in the long term.
Developers: We can use your help.
In my experience as a systems engineer, the biggest problem we've had with virtualization is that too many people who don't understand it well view it as a magic wand that you can wave to make all your capacity & provisioning problems disappear.
"Hey! We need a new server to run Blah version 3.0!"
"No problem! Sammy can create a new virtual server!"
"Oh wait - my bad. We actually need a whole farm."
"That's okay, he can whip up a whole batch of them!"
Ad nauseaum. About the worst I've heard was a clueless manager asking me if the resource requirements for Oracle 10g could be relaxed because we were running it on VMware. I actually found myself calling a "come to Jesus" meeting in which I explained, in as simple terms as I could, that "making the system virtual" doesn't mean that hardware requirements go away. Very, very few applications get faster when you put them on equivalent hardware, only virtualized.
My company works with several shops that are working on large-scale virtualization and common platform projects. I would say the biggest single issue is simply politics, because much of the initial work is affecting older platforms that are the biggest win technically and financially to replace. For instance, one shop has a significant investment in Alpha systems, and still has production servers that are 15+ years old running a huge chunk of their revenue producing systems. The folks working directly on the Alpha servers have considerable clout, since they've been the golden children for many, many years. Their bosses know how to play politics, and, considering that Alpha/VMS experience is one of those IT areas where there is little new blood from younger IT staff members, they are quite adept at finding reasons why it won't work to serve their own ends.
Not only that, but virtualization will result in lost jobs at some point. Many IT staffers are afraid, whether rightly or wrongly, of losing their jobs. In a sense, they are outsourcing a good chunk of their day-to-day duties. I remember when this particular company went to SAN's over the last half-decade, and you would have thought, from the way the Alpha guys were fighting it, that the world was ending. They created road-block after road-block about how they wouldn't be able to keep the systems running, how it wouldn't work in "their" environment, etc, etc.
And, because of the compartmentalization that often occurs in large enterprise, many of these guys have very little idea about anything outside their own box. I know guys who have architected corporate platform migrations who are so narrow in their focus that they have *NO* experience outside their box, be it a particular OS, a server type, a network type, whatever. When the box becomes a cloud of equipment, they are lost and often have little or no ability to work with the other layers involved. Learning new troubleshooting skills in these environments is a painstaking process, and not one that many people are comfortable with.
In the end, these various factors are creating far larger artificial roadblocks for implementing virtualization than any technical challenges. To top it off, much of this is being driven by financials. The CFO and CTO are desparately trying to find ways to cut costs. By the time this message percolates down to the workers, they feel threatened rather than empowered, and have little incentive (and generally no training, either) to be complicit in what they feel is a threat.
Bill
This is a problem with management and/or the IT staff.
Management should run the company in a way that cooperation is rewarded not punished. Consolidation to save money shouldn't result in harm to those who are making it happen or anyone else for that matter.
The IT staff as well as all of the other employees and officers should have the attitude that if it's good for the company and not bad for anyone else it's the right thing to do.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
VMware has multiple ways to balance and protect resources. You can set hard limits on VM resource utilization, ensuring that one machine can never take over a certain percentage of CPU, memory and even network bandwidth. VMs can also be given "shares", which determine priority over resources. In a contention for resources, the VM with the highest number shares is given immediate access to what it needs, with the lower share VMs splitting what's left over. This is the recommended way to handle it, as it gives you the best overall hardware utilization across your entire implementation.
Starting in VI3, VMware also introduced the ability for VMs to migrate automatically across an entire farm of hosts, based on server load. In my experience, with very little tweaking, VMware does a very good job of fairly balancing resources.
Items that need to be redundant, should not be virtualized on shared hardware. I've heard people want to virtualize redundant instances of directory services, databases, proxy servers...etc. I call this the "putting all your eggs in one,central-point-of-failure, hardware basket".
If you're doing something stupid like putting clusters or redundant servers on the same virtualization host, then I would agree. High availability loses it's meaning if all your nodes have a single point of failure.
However, there's absolutely no reason you can't make your virtualization implementation highly available itself. Right now, I have clusters running in VMware VI3, that are running on separate hosts. Even with DRS, which balances all your VMs across an entire pool of servers, I can ensure that redundant servers and clusters don't end up running on the same piece of physical hardware. And when you add HA into the mix, you also provide a level of high availability to systems that you might not otherwise have been able to justify the expense on.
On a go forward basis, you need step up to the plate and reach out to someone to get your thought processes in alignment. Perhaps a little thinking outside the box or brainstorming session would help to get someone to take ownership of this problem. Anyway, thanks for running this up the flagpole but lets take this conversation offline. I'll touch base with you later to discuss some of these basic action items.....
An effect I've noticed many times is that when you ask IT staff to vote, the windows IT staff always outnumber the Unix and mac It staff. Thus one man one vote favors the windows firedrill fix-it jockeys over the more talented kernel of Unix and mac support gurus. Yes I realize that's ripe for flamebait, but it's actually true. By and large windows has so many problems to keep functioning it lakes a large staff of low paid trained monkies on hand. The revenge of the c-strudents is that they out number the A-students who run the linux servers.
You want to watch a fight? Get the Windows Server sysadmins and the UNIX/LINUX sysadmins and ask each group which server OS should be the "Native" operating system under which the other runs....fun...
Nobody should EVER virtualize a server that needs special hardware. EVER. Virtual servers should be reserved for the most generic of hardware requirements. Once you start bringing in fax boards you need a dedicated physical solution. If you want to test that kind of thing, go ahead and virtualize it, but the production box should be physical. I shudder to think of a virtualized firewall or router. Ouch.
I don't know about you, but my servers run on the power of cotton candy and happy thoughts. -Anonymous Coward
This is a classic sign of a broken IT department. One place I worked, if you (well, if I) needed to increase the size of a database table, I had to get sign-offs from
net result? nothing ever got agreed. The simplest changes took forever and cost a fortune. The operation is now outsourced.
Who's to blame? Probably not the techies, they just pressed buttons. Quite likely the team-leaders for turning it political, definitely the IT managers who allowed the situation to continue.
Who kept their jobs?
yup, the managers! You've been warned: infighting only hurts the foot-soldiers, the generals aren't affected. Sort it out yourselves or you'll have to start learning chinese.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
But the IBM commercial told me that i could replace my entire datacenter with a single server! Are you telling me that those two youngish looking racially diverse guys having a conversation at a coffee shop about sloshing hundreds of thousands of dollars worth of hardware around even though neither of them really seems to be sure what all of it does were LYING TO ME!?
NewslilySocial News. No lolcats allowed.
Many large virtualized deployments include very advanced technologies such as shared SANs, shared infrastructure, and complex virtualization tools.
Frankly, I would argue that you are probably just redeploying people resources into different roles and responsibilities, while probably saving on hardware and energy costs for the infrastructure through consolidation.
Lindsay Blanton
RadioReference.com
Office Space, meet Slashdot.
- They said I'm allowed to troll at a reasonable volume.
- Now Milton, don't be greedy, let's pass along the karma and make sure everyone gets a piece.
- Excuse me? Excuse me, senor? May I speak to you please? I asked for an overlords joke, and they brought me an in soviet russia joke.....and I said no trolling, NO trolling for the replies, but it lots of trolls, big stupid trolls
- Oh, and remember: next Friday... is MS bashing shirt day. So, you know, if you want to, go ahead and wear an MS bashing shirt and jeans.
- I can't believe what a bunch of nerds we are. We're looking up "social engineering" in a dictionary.
- Just remember, if you hang in there long enough, good things can happen in this world. I mean, look at me.
- Looks like you've been missing a lot of work lately.
These posts express my own personal views, not those of my employer
We recently moved everything into virtua-land, complete with a hige SAN, fiberchannel switches, blade servers - the whole nine yards.
While I do think the move was a net positive, the complication of 60 physical servers was more or less replaced by the complication of all the new SAN/Bladecenter components and their interdependency.
One particular thing we've run into is "firmware hell", where you have several components in the chain that all require firmware updates and all depend on each other.
I don't always use unix-like operating systems; but when I do, I prefer FreeBSD.