Slashdot Mirror
The Trouble with Virtualization - Cranky IT Staffs
lgmac writes "A new survey on the results of Enterprise use of virtualization shows that the process is seeing wide and appreciative use. Technical hurdles are obviously the biggest problem facing corporate IT shops. Just the same, political squabbles among IT staffers fighting for turf after being forced to work together in new ways seems to be a going concern as well. 'Technical woes rank higher--to be expected when CIOs deploy a new technology such as virtualization. However, the politics pain many of you. Remember, virtualization not only asks people to cede some control over their physical server kingdoms, but also asks IT experts from different realms to work more closely together.'"
34% of surveyed companies have been running virtualized desktops? Putting aside that that number doesn't seem to square with the "Virtual Desktops a Hard Sell" table below, does that seem likely?!?
What I'm listening to now on Pandora...
Unfortunately, the story really doesn't say anything of value. Just lots of filler, and an excuse to throw ads at you. Next, please.
Who would have thought consolidating servers would require less staff? If your company is thinking about doing Virtualization, make sure you are in charge of what remains of the pie!
In Soviet Russia, articles before post read *you*!
My companies biggest problem concerning virtualization at this point has been backing up running copies of virtual server without interruption, anyone have some insight on this?
Technology is continually changing. Those who adapt will be the most successful. Those who don't will eventually be pushed aside. Fighting over turf won't get you far in a corporate environment in the long term.
Developers: We can use your help.
In my experience as a systems engineer, the biggest problem we've had with virtualization is that too many people who don't understand it well view it as a magic wand that you can wave to make all your capacity & provisioning problems disappear.
"Hey! We need a new server to run Blah version 3.0!"
"No problem! Sammy can create a new virtual server!"
"Oh wait - my bad. We actually need a whole farm."
"That's okay, he can whip up a whole batch of them!"
Ad nauseaum. About the worst I've heard was a clueless manager asking me if the resource requirements for Oracle 10g could be relaxed because we were running it on VMware. I actually found myself calling a "come to Jesus" meeting in which I explained, in as simple terms as I could, that "making the system virtual" doesn't mean that hardware requirements go away. Very, very few applications get faster when you put them on equivalent hardware, only virtualized.
virtualization is a blessing, for me at least. In the past I had to run a couple of servers at home to do my work. But now, I have an ubuntu Host, and I run everything from there, clients, servers, everything . That saves a lot of hardware headaches and backup problems, all it take it to copy a couple of file.. It was a tad tricky to setup, but when it was up everything was running great. With todays cheap Storage, RAM and Processing power, this is truly a blessing.
I'd imagine that one of the big problems with virtualization is clueless IT managers/staff who don't understand that you basically are dividing a server down into sub-servers. I've encountered a few people who seem to think that virtualization multiplies the server resources. That is, everyone using a VM basically gets the full specs of the host machine--all at once! Ugh! Maroons!
My company works with several shops that are working on large-scale virtualization and common platform projects. I would say the biggest single issue is simply politics, because much of the initial work is affecting older platforms that are the biggest win technically and financially to replace. For instance, one shop has a significant investment in Alpha systems, and still has production servers that are 15+ years old running a huge chunk of their revenue producing systems. The folks working directly on the Alpha servers have considerable clout, since they've been the golden children for many, many years. Their bosses know how to play politics, and, considering that Alpha/VMS experience is one of those IT areas where there is little new blood from younger IT staff members, they are quite adept at finding reasons why it won't work to serve their own ends.
Not only that, but virtualization will result in lost jobs at some point. Many IT staffers are afraid, whether rightly or wrongly, of losing their jobs. In a sense, they are outsourcing a good chunk of their day-to-day duties. I remember when this particular company went to SAN's over the last half-decade, and you would have thought, from the way the Alpha guys were fighting it, that the world was ending. They created road-block after road-block about how they wouldn't be able to keep the systems running, how it wouldn't work in "their" environment, etc, etc.
And, because of the compartmentalization that often occurs in large enterprise, many of these guys have very little idea about anything outside their own box. I know guys who have architected corporate platform migrations who are so narrow in their focus that they have *NO* experience outside their box, be it a particular OS, a server type, a network type, whatever. When the box becomes a cloud of equipment, they are lost and often have little or no ability to work with the other layers involved. Learning new troubleshooting skills in these environments is a painstaking process, and not one that many people are comfortable with.
In the end, these various factors are creating far larger artificial roadblocks for implementing virtualization than any technical challenges. To top it off, much of this is being driven by financials. The CFO and CTO are desparately trying to find ways to cut costs. By the time this message percolates down to the workers, they feel threatened rather than empowered, and have little incentive (and generally no training, either) to be complicit in what they feel is a threat.
Bill
It may be more practical to back up the system from within the VM, i.e. treat it as if it weren't a VM. By definition this will be on a live system.
Another option:
Have your VM use a checkpoint disk. Once a day shut down the VM, merge the changes from this week into the checkpoint disk, and restart the VM. This may take anywhere from a few minutes to tens of minutes. Restart the VM. Back up the checkpoint-disk image.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
[Virtualization] also asks IT experts from different realms to work more closely together.
Oh yes, there will be blood.
Dedicated Cthulhu Cultist since 4523 BC.
How do you ensure that the VM supervisor fairly and efficiently allocates resources to the VMs? The mainframe people put a great deal of work into this area. One badly behaved VM shouldn't be able to degrade the performance of the other VMs.
Mea navis aericumbens anguillis abundat
This is a problem with management and/or the IT staff.
Management should run the company in a way that cooperation is rewarded not punished. Consolidation to save money shouldn't result in harm to those who are making it happen or anyone else for that matter.
The IT staff as well as all of the other employees and officers should have the attitude that if it's good for the company and not bad for anyone else it's the right thing to do.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
At my job, we honestly don't have a valid reason to adopt virtualization right now. It'll actually cost us more money to accomplish the same job we're presently doing without it. But my boss wants to deploy it somehow only because it's one of the latest buzzwords. I guess it looks good to have some vm experience on my c.v. also ;-)
Yes, well, naturally the problem is us "peons" can't work together. It has nothing to do with the fact that our bosses don't have a fucking clue about how to use the technology.
Yes, I'm the cranky IT guy. I'm responsible for the stability, scalability, and security of our IT resources. Every time I hear about someone pushing virtual servers it makes me want to jump out of a window.
Items that need to be redundant, should not be virtualized on shared hardware. I've heard people want to virtualize redundant instances of directory services, databases, proxy servers...etc. I call this the "putting all your eggs in one,central-point-of-failure, hardware basket".
Virtualization has its place, but thanks to falling hardware costs, sometimes it is worth dedicating small, cheap, boxes to a specific task.
-ted
It's a spam troll.
The problem is not that we don't want to work together. It is that you often cede control when you virtualize. And most of us don't love giving up control.
With virtualization in some Corp's, you have to ask for another of the 32 processors, instead of just having the headroom all the time.(work that one through a buricratic organization, it can take months)
Say you have a need to add another fax board(or whatever) to the virtualized x86 server, to find that they stuck some mission critical Virtual Environment on the Server and It CAN'T come down for another 2 weeks.
Yep, it saves hardware, but multiplies headaches in some situations. It is no wonder some fear it.
Cranky IT Staffs, All IT Staffs are cranky. What of it. Long hours, work weekends, holidays, 7/24/365 on call schedules. Suck it up whimp.
but the reality is that if you're not responsible for something important, even mission critical, then you just let someone paint a layoff target on your back. that's the political reality of many companies. It's easier to fight to keep what you have than it is to expect that someone will give you something good to manage after you give up control over what you have right now. Virtualization is actually a poor example of the cross functional integration in many companies, since it's squabbling within the IT department. Try implementing an ERP system (with IT, Finance, and Sales fighting over ownership).
An effect I've noticed many times is that when you ask IT staff to vote, the windows IT staff always outnumber the Unix and mac It staff. Thus one man one vote favors the windows firedrill fix-it jockeys over the more talented kernel of Unix and mac support gurus. Yes I realize that's ripe for flamebait, but it's actually true. By and large windows has so many problems to keep functioning it lakes a large staff of low paid trained monkies on hand. The revenge of the c-strudents is that they out number the A-students who run the linux servers.
You want to watch a fight? Get the Windows Server sysadmins and the UNIX/LINUX sysadmins and ask each group which server OS should be the "Native" operating system under which the other runs....fun...
"when CIOs deploy a new technology"
That could be your problem right there. When a specific technology or whoop-do-doo product is pushed from the top down, rather than the bottom up, it's a problem. That's not the same as management saying "Get this done", so much as it's "Use this fancy thingy I read about in the newspaper... who cares what it does or if there is something better, I'm the decider!"
For no readily apparent reason, my company started leasing Windows servers a decade or so ago instead of buying 'em outright. The good news is you have new hardware every three years. The bad news is you have to move everything.
A couple years back they went virtualized with everything. Now lease-rolls are a piece of cake; shut off your virtual server, zone the SAN storage so the new box can see it, and fire it up on the new box. Poof.
That said, I'm still glad I'm not a Windows admin here. Who leases servers?!
This is a classic sign of a broken IT department. One place I worked, if you (well, if I) needed to increase the size of a database table, I had to get sign-offs from
net result? nothing ever got agreed. The simplest changes took forever and cost a fortune. The operation is now outsourced.
Who's to blame? Probably not the techies, they just pressed buttons. Quite likely the team-leaders for turning it political, definitely the IT managers who allowed the situation to continue.
Who kept their jobs?
yup, the managers! You've been warned: infighting only hurts the foot-soldiers, the generals aren't affected. Sort it out yourselves or you'll have to start learning chinese.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
With every decision that you make: Is this good for the company? Am I helping the best way that I can for the company...
One thing - CPU HOG that virtualization really fail. Our users just end up turning up their nose
after finding out VMWARE is a pig and a couple big compilation really kill the performance of the system
So we end up buying more machines and dumping VMWARES....
What I've found to be a pain is when people start running testing environments with like 7 servers in bridge mode with static IP#'s in the DHCP pool because they don't know any better. Then IT trouble tickets come in asking why people are getting IP conflicts and interrupted SSH connections to SVN servers while no IT trouble-tickets come in from QA as their invalid network configuration changes are distracting them entirely with test results that are randomly terrible and they just can't seem to figure out why.
Just as with physical machine deployment, virtual machines have to be planned. As long as that is taken care of things seem to be OK. At least in my experience.
Our good friend, Mr. T., needs to pay them a visit to talk about the DOs and DON'Ts of Virtualization...
[not an endorsement for the advertised product--it's just ridiculously funny]
I might know what I'm talkin' about, but then again, this is Slashdot...
"A new survey on the results of Enterprise use of virtualization shows that the process is seeing wide and appreciative use."
... is that like the cloaking device?
Virtualization
... who's the c-student again?
We've had the same issues with bladecenters and their integrated switches (LAN and SAN). The serverteam procures the bladecenters, and all the components inside. They then consider it their turf to manage the embedded switches. It turned into a political nightmare to wrest control of those switches away from them.
It's happened twice to me at two different companies.
Whenever I need a machine scratch-pad, I boot up a VMWare machine. Test the software or do whatever I need to do and delete it. But while it's running, it broadcasts itself on the local net. Admins really freak out when a machine named //FAKEOUT or //BOGUS suddenly shows up on their net.
I've given two different IT guys at two different companies cardiac events over it.
Sorry, fellas.
Weaselmancer
rediculous.
The REAL trouble with virtualization is that it solves a problem that should never have existed in the first place. Which problem is that? Windows servers can only run one application. Back in the NT 4.0 and Win2K days many of us learned, often painfully, that Windows servers could not run a mixed application load and could not run at or near capacity. This is not true of other platforms. For that matter, it may not be true of modern Windows servers.
Virtualization is wonderful for development and testing environments but it should not be necessary for a production environment.
Yup, another metacity link.
We need a mod down tag that just plain says "Spam"
It seems that 90% of the problems are not technical at all but social. Turf wars, pissing contests, narcissists, clueless managers, clueless software engineers who none the less insist they know it all, newbs who think reading a book on Rails means you owe them 70K a year, technical staff members who expect to be 'lead by the hand' when learning new technology etc.
Is it me or does this seem to pervade IT more than other fields? And if so, why?
putting the 'B' in LGBTQ+
I'm an IBM sales rep. I'm currently in storage sales, but I started out cross brand, and we did alot with virtualization.
VMWare and virtualization aren't awesome because they save you on hardware costs. They merely make sense in SOME instances because you can save a metric butt-ton of money on power, cooling, space, and more importantly: software licensing fees and hardware maintenance costs.
If you virtualize 50 small intel servers on a few large boxes, you see a significant savings on the maintenance fees for those boxes. But more importantly, on some specific pieces of software (Oracle, I'm looking at you), where the pricing is on a per processor basis, it's possible to save hundred's of thousands of dollars on software licensing fees.
Presuming that the hardware and vmware are setup properly and in a way that makes sense, you can also realize some extra ease in managability. But that tends to be related more to how the company's IT department is set up.
As other's have pointed out, it's also a fun test environment.
Many large virtualized deployments include very advanced technologies such as shared SANs, shared infrastructure, and complex virtualization tools.
Frankly, I would argue that you are probably just redeploying people resources into different roles and responsibilities, while probably saving on hardware and energy costs for the infrastructure through consolidation.
Lindsay Blanton
RadioReference.com
When I was working for the largest school district in Colorado we got a new CIO. She did not show up at work for the first 6 months do to medical reasons and then came in sporadically. She began pushing out all the male veterans (mainframe types with 20+ years of experience). She replaced them with her friends and neighbors all at 6 figures plus and no IT experience. One of her first initiatives was to replace are old sun mail server that had cost less than 15k over ten years and was very reliable with a 4 million dollar installation of exchange. This because she did not want to use a mail client that she was not familiar with. The next thing she did was write a glowing review of her "revamping" of the district IT department. CIO Magazine printed it with out as much a question to anyone else about weather or not it was true. She has since been demoted due to her many failures and CIO Magazine has not printed anything to update their story. Anyone can write a "story" for this mag and they will print it with no questions asked. I won't even waste my time reading a story from such a rag.
At the far end of the spectrum is a CIO who totally loves virtualization and believes that everything should be virtualized in order to increase ROI.
Will he take blame when his critical applications are more difficult to support, because he thought it would be a good idea to virtualize those, too? Will he make the connection when his MTTR goes up because of the added complexity he's created in the IT department?
I like how they painted all push-back as 'politics' with such a broad scope. Things are so much simpler when you try to operate a datacenter from 10,000ft, and are able to categorize the problems in ways that support your own decisions.
Anytime you end up arguing something like this, make sure you provide the first working physical solution in the background ASAP. The first one working w/backups and all is the one that gets approved. IMX, you piss off ppl less, too, when you don't start a verbal nuclear war, but just rather get the job done. That way you can act nice towards the other party while still getting your way. The resentment that this path produces is still less than forcing your will when everyone's dug deep into the trenches. Yea, the first rule of politics-driven development is to be first w/the actual prototype. That tends to beat the politics, as the ppl making decisions rarely care about anything else than money, and that's why you have to come up w/tangible proof of what you're saying. That's understandable.
The lyf so short, the craft so long to lerne
The one thing I have been able to rip from users is certain services. Like an oracle/mysql/postgress server. In the past the users felt they had to maintain it. Now we have one server and they use it as a service. The cluster handles keeping it up. This only works well with RedHat and only if you know what you are doing. Now the end users are relieved. They don't have to worry about the database, server configuration and maintenance that used to dog them with Solaris, Windows, BSD and SUSE boxes. Windows being the biggest PIA because Microsoft does things to you if you update it. Then the other issues the /. crowd is used to.
To me I have a load balancer that is managed by a gang of web servers as a clustered service so it never goes down. The web servers are highly available so I can reboot whenever I want. The database is also highly available. People just upload stuff to a virtual address and a different port and it is just there. It gets updated very quickly when a patch comes out. In short I don't have to even schedule down time anymore unless we have a power outage. Just be sure you have a place to test updates first. If something goes wrong with the clustering software, it can really go wrong. Then it is like having 100 dishes up in the air. Instead of dropping one dish, you drop 100.
The thing I hate about it is trying to explain it to end users and even guys I used to think were technical. They just don't grasp the concept of a gang of servers, virtual servers and virtual databases. They think that if someone gets a form from one machine, it must return the data to that machine. As of the server is like a logon session. Maybe it is that "logon to www.sitename" bullshit they put out there in the news. They should say "visit site www.sitename", leave "logon" out of it entirely. Anyhow, eyes gloss over and it's a bitch to get them back. Sometimes now I just tell them we are moving them off of their old machine and let it go at that. They don't have a need to know. MUCH easier that way. The only PIA is when they ask what the serial number is of "their" machine.
Still, there are some people that just don't want to give anything up. I do agree that this environment requires more cognitive abilities from the IT staff. I don't think you can be average and get by anymore. The IT staff needs to have bright people now. People that can learn. Otherwise they are left behind and it can be brutal.
We recently moved everything into virtua-land, complete with a hige SAN, fiberchannel switches, blade servers - the whole nine yards.
While I do think the move was a net positive, the complication of 60 physical servers was more or less replaced by the complication of all the new SAN/Bladecenter components and their interdependency.
One particular thing we've run into is "firmware hell", where you have several components in the chain that all require firmware updates and all depend on each other.
I don't always use unix-like operating systems; but when I do, I prefer FreeBSD.
Our virtualization setup is pretty simple, so it hasn't yet become a huge squabble; but what runs on top of it is looking likely to turn into a major turf war. The office that I work with is a midsize(ish) educational environment. c. 1000 desktops across 8 sites, with a bunch of servers, fiber between municipal buildings, and VOIP. Mostly Windows, with some VMware ESX on the servers. In staff terms, there is a big divide between desktop and server people. The server guys have control over the servers, the AD configuration, and the various admin tools for switches and firewalls. And they like it that way. The desktop people don't have nearly as much power; but they are the ones most directly responsible to users(they build all desktop images, and support users).
There has always been some resentment over access to the cool tools and toys; but the thin clients are set to really raise hell. With our Citrix setup, we suddenly have some hundreds of users whose desktops are determined by the server guys, not the desktop guys. Guess who still gets to support those users? The punchline, longterm, is that the server guys are going to start having to care about desktop experience, or they are going to have to give the desktop guys enough power to care for them. Not sure which it will be; but it isn't either yet.
It's starting to happen again - soon a cluster will not be seen by newbies as not a lot of physical machines but as a lot of applications in userspace pretending to be hosts and we'll have trouble dealing with those newbies. Don't fall into the marketing trap - call them virtual hosts and virtual clusters or we'll fall into the trap of those idiots that call the beige box on the desk a "hard drive".
Servers shouldn't be in ip subnets that are utilized for DHCP clients! What moron has DHCP scopes that cover his IP ranges for his servers? What? reservations you say? Who cares, it's just plain retarded.
If the vmware environment is a test environment, it should probably be in its own vlan too.
Foul for failing to use 'paradigm' at least twice. The penalty assessed for said violation is a mandatory reading of 5 whitepapers from www.networkworld.com. And for the record, CXOs don't use technical jargon such as 'offline' - that kind of talk is for geeks and communists.
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
Slightly OT, but I've gotta' ask, what are you guys running on your SAN for concurrent file system access? GFS? I've got an ISCSI setup at home and I love it, but I'd really like to have a target mounted by two initiators at the same time (on separate subnets). I've read that GFS, like most concurrency technologies, is fairly slow - but it seems to be the only 'mature' solution ATM. Any suggestions or insights from your recent upgrade?
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
Are you using LVM2 or ZFS? Both have real time snap-shotting. You can rebind the snapshot read-only in a shared directory and send it upstream while the disk is still being used. NFS might not be ideal with oplocks and such, or any byte range locks, for that matter. Of course, if you have any database access going on, it might be hairy. Also, call sync once or twice before you snapshot!
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
Arg, all Windows shop limits clustering ability quite a bit from what I understand. I don't know much about Windows servers, but I think file locking is the issue, IIRC. POSIX file locking has a stupid implementation where if you repeal a single lock, you repeal them all. FWIW, you might want to try CentOS 5.1. It has GFS clustering support, fencing, and Xen (software or full if your processor has the extensions) kernel hypervisor, out of the box. Not to mention iscsi enterprise target and initiator. Also, samba 3.0.28(7?) introduced clustering abilities. I haven't played with them extensively, but they seem like really cool toys :). I'm not sure why they freaked about the BSD (I think there's a new dragonfly, BTW) VM - it was probably the most secure platform in the farm! Hope some of this was helpful. Good luck! :)
If I mod you up, it doesn't necessarily mean I agree with what you've said, sorry.
I'm posting this from a virtualized desktop and it truly sucks.
threadeds blog
"Suck it up you Nancy!"
Quote from http://www.3vista.com/
There is a hidden cost with virtualization. Unless your backup system is capable of data deduplication be sure to include the added burden on your backup infrastructure when you have to save all of your operating system files many times more than you currently do. Oh and include the cost of memory overhead for each virtual OS. Don't be dazzled by the virtual OS promise and forget your basic TCO calculations.
Also you want to watch out for creating a single point of failure with the underlying (shared) file system. You can take down a lot of systems by making one mistake at the shared FS level once you virtualize. You insulate yourself from a rogue application or crashing operating system by creating a bigger single point of failure further down the stack (at the file system/SAN level). Is that a good thing for production systems?
Good points you've raised.
:)
My solution would be to weed out the server guys with no social skills and promote the desktop guys with brains. Not easy to find either in some environments I know....
disclaimer: I'm a network guy
In college I set up a machine that had various virtual machines on it, each of which had a vulnerability to be exploited by class mates in my network security class.
It was a humble machine - Duron 900 with a gig of ram, I think - but it ran 9 VM's (each with 256 MB or so of RAM) on VMWare Server concurrently and did well because of intelligent swapping (and a huge swap partition). Mind that the actual traffic on these machines was very low and I'm sure it would tank in heavy loads, but for my low traffic needs it sure beat setting up 9 different computers.
Now if we were under the expectation that it would perform as well as 9 Duron 900's each with 256 megs of ram we would be lunatics.