Slashdot Mirror
Ask Slashdot: What Type of Asset Would You Not Virtualize?
An anonymous reader writes "With IT and Data Center consolidation seemingly happening everywhere our small shop is about to receive a corporate mandate to follow suit and preferably accomplish this via virtualization. I've had success with virtualizing low load web servers and other assets but the larger project does intimidate me a little. So I'm wondering: Are there server types, applications and/or assets that I should be hesitant virtualizing today? Are there drawbacks that get glossed over in the rush to consolidate all assets?"
Shared disk does not make I/O happy.
Working...
Virtualize management.
Have gnu, will travel.
Assets not to virtualize:
1) Women
2) Beer
3) Profit
"Our two-party system is like a bowl of shit looking at itself in a mirror." - Lewis Black
ibm even says give it its own physical machine if you're going to virtualize it.
The company cafeteria isn't all that great, but jeez nothing is less satisfying than a virtual burger and virtual fries.
Sig Battery depleted. Reverting to safe mode.
Gold, houses, aircraft carriers, 17th century Dutch paintings.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
How about backups?
Consolidating and virtualizing your backup servers sounds like a recipe for trouble to me.
Dan Aris
Fun. Free. Online. RPG. BattleMaster.
I would not virtualize the servers that are running the virtual machines.
Do you sense a growing suspicion that Facebook has been counting virtual revenues...?
Well yes Databases would make a poor virtualization target. Also your heavy memory usage java app like the company app server using a terabyte of ram to display the department wiki.
Got Code?
Seems like a front-end serving statically cached content is a great match for Virtualization. DB servers and search servers (Solr, etc) aren't a good match imho, unless you have a very well implemented sharding/horizontal scaling solution. If you pre-generate your content, we've typically used hard boxes for those as well, but you may benefit from virtualizing those if you want to easily scale horizontally (assuming you have the hypervisor overhead).
Having DNS virtualized can make it hard to start vCenter. Also, you probably don't ant to virtualize your vCenter SQL backend.
Don't virtualize anything requiring tight scheduling or a reliable clock, such as a software PBX system performing transcoding or conferencing.
http://www.vmware.com/files/pdf/Timekeeping-In-VirtualMachines.pdf
Pretty much anything can be successfully virtualized if you throw enough hardware at the host. Just keep in mind that these machines are all actually running on the same processors, and there's probably going to be a way to escalate rights from VM to host or VM to VM. In your environment this may not be an issue, but it's worth keeping in mind.
Yes, having the branch servers (domain/user profile) centralized saves on server hardware and maint, but that gets eaten up the first or second time a subcontractor has to wait 6 hours for a profile to load over a satellite link. I know. I'm that contractor.
The only workloads that you can't really virtualize tend to be things like OS/400, but that is where things like LPARs can come in OR workloads that do a lot of privileges calls to the CPU or a specific instruction set. Also there are a slew of non-technical reasons I've seen like in Healthcare or Pharma where a specific machine is written into a specification for drug manufacturing or such.
Even still there really aren't any workloads you can't virtualize and realize some sort of benefit from. Even those with high requirements for CPUs or Memory hogs can be virtualized. It's not always the best financial decision from the hardware cost and licenses, but most organizations benefit then from the ability to move the VM from one physical host to another to avoid downtime and easier backups. Having worked on a lot of the largest VMware deployments out there, I've yet to find an application we can't virtualize for some gain in performance (newer hardware) or get better resiliency (less downtime, easier backups, HA features outside of the OS).
I can only please one person a day. Today is not your day, and tomorrow does not look good either.
- Telephony and rea-time voice/video servers (Asterisk for example). You don't want to explain to your big boss why his phone lines are having hiccups :) Preferable to have an old Pentium running that one somewhere.
- Real-time servers, like game servers (Minecraft), as they constantly take up resources.
- NTPD servers (Network Time Protocol). Worst case, run it at layer 0 (host machine). But not VM
If you look at the pattern, these are all real-time services that have very little leeway for latency. Yes, it's possible to virtualize them, but it's more work than it's worth, really.
For me, IMHO, everything else is fair play. Had good luck with Windows Active Directory servers (yes even PDC), SVN, web services, file servers, databases, memcache, and many others.
Makes little sense to not run on metal when using an HPC. I can understand the benefit of being able to better utilize the hardware you have, as well as the potential lessening of your datacenter footprint in space, cooling, electric, etc. but when you are dependent on having quick(and ever quicker) turn around times because of business needs, it hasn't been my experience that the cloud makes sense, at least in production environments. Granted, for Dev & QA HPCs, go for it, but not for production.
Most things you can virtualize, even if you only have the 1 VM on the machine it does allow for easier migration between physical hardware (especially if Windows based).
Heavy I/O tasks you might not want to, if you do, local/DAS dedicated storage is preferred, if you virtualize these, it quite possible you want to have 1 VM to the 1 Physical machine, it doesn't save you space but as stated above can make things easier for migration between hardware.
Management and Monitoring you can virtualize but do so with local storage, not network based (iSCSI, AoE, NFS). If you must use shared storage make sure you have 2 on seperate arrays.
The backup server is OK to virtualize, but we keep the backups on local storage and tape, not on the shared array - the reason is the same as with the managment and monitoring not being on the shared storage array. If it goes down so you lose all machines operating off that array.
except BIG backend databases and keep your DBs separate from any disk groups used for virtualization
The best test environment is production. - Me
chrome://browser/content/browser.xul
I have seen the time on virtual machines hopping around -- even those that are running ntpd.
The real "Libtards" are the Libertarians!
My suggestion, based on past experience, is to check with your third party vendors and make sure they will continue to support their software in a virtualized environment. I've run into a few who basically said that if their product wasn't running on real hardware than they wouldn't support it any longer. It doesn't happen often, but check just to make sure.
Let's take this one step at a time. What does virtualisation do? Well, it lets you split up one big, beefy system into a large number of smaller, not-quite-so-beefy systems. In the process, I/O becomes essentially random (because while host A may be reading sequentially from its disk, and host B may be reading sequentially from its disk, and host C may be reading sequentially from its disk, that means that the disk - if those virtual disks are coming from the same spindle - is jumping from A to B to C to A to B to C to A to B to C, and performance is going to go through the floor. Remember this story? Yeah, that's exactly what I'm talking about.)
So anything that has high CPU requirements, or high I/O throughput requirements, probably shouldn't be virtualised. Anything with high RAM requirements might be a candidate for virtualisation alongside low-RAM applications, if the CPU and I/O needs are low, as long as the physical machines running the virtual hosts have an imperial buttload (that's the technical term - the metric buttload is smaller) of RAM. Everybody's situation is different; you'll know this better than anybody on slashdot.
On the contrary, these instructions are run directly by the host processor at full speed.
VMware emulates the rest of the computer, but it's just time-slicing the CPU to the emulator, so pure CPU runs more or less at full speed.
Please note in your virtual machines that the CPU is not virtualized, it is the same make and model as your actual hardware.
It's I/O that suffers with virtualization, because the I/O devices have to be emulated.
The small shops are probably more fearful of virtualization, because they're small. A bigger company has a better chance of recovering from a screw up than a small one during the transition. I might be completely wrong, but that's my take on it.
--wmbetts
Any heavy number-crunching will suffer from being virtualized.
I've not seen any hard numbers (that I can remember) but given userspace code runs directly on the CPU, with hardware-assisted memory translation, in most modern hypervisors - I'm not sure why this would be the case?
Food and water.
File under 'M' for 'Manic ranting'
Most database servers are already doing the same things that virtualization accomplishes. SQL Server 2012 as an example can support multiple database instances, each with multiple databases and will use every last resource available, and be more efficient than hosting multiple copies of in their own OS instance in VMWare.
Along with virtualization management, do not virtualize whatever system provides authentication to allow you to manage the VM's (domain controllers in the case of hyper-v). Other than for hardware requirements (PBX, phone interfaces,anything requiring proprietary hardware) everything else is fair game. The list of hardware that can not be accessed by a VM is getting shorter, as virtualization giants have started to support giving VM's some GPU time on the host, as well as access to USB devices.
Firewalls, load balancers, anything that is typically hardware with dedicated ASICs. These virtual appliances typically are very taxing to your virtual infrastructure and cost about the same as the physical box. Also, iSeries, but I'm pretty sure I could do it if our admin went on vacation for once :)
Imagine coming up from a stone cold shutdown. What would be a super thing to have? How about DNS and DHCP? AD too if that's your thing. Some nice little box that can wake up your LAN in 5 minutes so you can start troubleshooting the boot storm as the rest of your VMs try to start up and all get stuck in the doorway like the Three Stooges.
Acquiring and using sockets is high overhead for the VM and can become a bottleneck. We ran an experiment on virtualizing our cache servers, and with fairly identical specs the real machines had 25-50% better throughput when the machine was under load.
If you're going to virtualize something that gets a lot of traffic then it makes sense to scale up the server and environment.
If you're talking about virtualizing an enterprise scale server/server farm then you'll want a solution that is designed to handle that sort of situation.
As some people said, shared disk doesn't make I/O happy. That's a key point which is dealt with in enterprise scale virtualization by spreading the load across many different systems. So the hit of shared load is mitigated by access to multiple systems with redundant information. There are some very cool products that do this sort of thing very well.
But generally it's a bad idea to shoe horn an enterprise system into a limited virtualized environment where performance will suffer.
You don't want to virtualize unless you're consolidating servers. The costs just don't make sense. Where you save the money is when you get three servers to do the work of 10... or 10 to do the work of 100.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
Virtual sex is just not the same.
You would hurt things by attempting to virtualize an HPC (High Performance Computing) environment. It's all about raw horsepower, and you sacrifice a chunk of that in a VM environment.
Depending on the environment and the available assets to the IT Department.
As an example:
Assume you have VMWare ESXi 5 running on 3 hosts with a vCenter and a combined pool of say 192GB of RAM, 5TB of disk, 3x1Gbps for NAS/SAN/iSCSI and 3x1Gbps for Data/connectivity.
It would become unwise in such an environment (without funds to expand it) to run any system that causes a bottleneck on your environment and thus decrease performance for other systems. This can be:
- Systems with High Disk load such as heavy DB usage or SNMP Traps or Log collection or Backup Storage Servers;
- Systems with High Network usage such as SNMP, Streaming services or E-mail;
- Systems with High RAM usage.
For this example, any of the above utilizing say 15% of your total resources for a single instance server would ultimately become cheaper to run on physical hardware. That is, until your environment can bring that utilization number down to 5% or is warranted/needed/desired for some reason.
In my environment, we have a total of 15 ESXi v5 hosts on Cisco UCS Blades with 1TB of RAM and 30TB of Disk on 10GbE. We do however refrain from deploying:
- Media Streaming servers
- Backup Servers
- SNMP/Log Collection Servers
Hope this helps!
The advantages of virtualization are too great to not do it whenever possible.
The only limiting factor is, really, is how much money you have to spend on your virtualization infrastructure. VMWare's licensing got a little nutty, and SAN storage got really pricy last year.
But it's worth it. Once you have a nice VMWare cluster running, SO many things become easier. And some things that were damn near impossible before become simple.
That said, you probably want to keep at least one domain controller and one DNS server as physical boxes. Makes maintenance of the VMs easier, if you ever have to reboot the whole thing.
Just don't virtualize everything into a single host. Have multiple hosts and set the virtualization management to fail over. Otherwise losing one server means losing all the servers. Then only make enough VMs so that if one host failed, things would just run slightly annoyingly slow on the one picking up the load until the problem is fixed. Of course, don't let the annoyingly slow happen to anything mission critical with tight response requirements no matter what.
You can actually virtualize a whole lot of things. The real key is to put a lot of money into the virtualization hosts. CPUs/cores, ram, a really good storage system.
For the small budget, you can get by on a lot less.
I have virtualized several entire development stacks (app servers, DB servers, storage servers, reporting servers). {But you trade a bit of performance for a reduced physical server count (10 or 15 to 1? A pretty good tradeoff if done right)}
You CAN virtualize SQL servers. Most business DB servers at the small shop end are fairly light load (like finance systems) and virtualize well. {But if performance makes you money (ie: you have a SAAS product - then stay physical }
You CAN virtualize an entire Cognos stack (it is made up of many servers depending on how complex your environment is). {However, IBM is correct that in a heavy BI workload environment deserves physical servers. I run over 18,000 reports a day on my produciton stack. Not going to virtualize that any time soon.}
You CAN virtualize entire profit generating websites. {As long as you keep an eye on host CPU and perceived performance at the end user level}
You can virtualize a lot of this in relatively small environments.
But.. Everyone here who has said it is correct: DISK IO is a major contention point. If you stuff all of your virtual machines inside one single giant data store (VMWare term for a single large disk partition) and expect super fast performance 100% of the time, then you will be greatly disappointed. One of my own stacks would grind to very intollerable performance levels whenever someone restored a database to the virtualized DB server. We moved that DB server virtual machine's disk load onto dedicated drives while leaving the machine itself virtiaulize, and all those problems went away.
Do not virtualize anything thar requires multiple CPUs (cores) to operate efficently. SQL Server is an example fo something that works better with multiple CPUs at its beck and call. In virtualization though, getting all the CPU requests to line up into one availabe window bogs the whole virtual machine down (jsut the VM, not the host). If your workload can't survive on a single virtual CPU, or two at most (single core each), then you are best to keep it on a physical server.
Time sensative systems and high compute workload processes are also ideally to be left out of virtualization. Except.. If you can put a beefy enough box under them, then you might get away with it and not notice a performance impact.
The biggest mistake made when going into virtualization (besides not planning for your DISK IO needs) seems to be over provisioning too many virtual machines on a host. This is a dark trap if you are lucky to have the money to build out a high availability virtualization cluster. You spread you load across your nodes in the cluster. Then one day one goes off line and that workload moves to another node. If you only have two nodes, and one is already over subscribed, suddenly the surviving node is way over its head and everything suffers until you get in and start throttling non esscential workloads down.
So, what do you not virtualize? Anything where performance is critical to its acceptance and succcess. Anything that a performance drop can cost you money or customers. (Remember that internal users are also customers).
Plan ahead ALOT. If you feel like your not going in the right direction, pay for a consultant to come in and help design the solution Even if it is only for a few hours. (No. I am not a consultant. Not for hire.)
I work for a telco and we've virtualized a lot of stuff, including telephone switches. We ran into a lot of problems with things that required reliable clock/syncing. 2 way video, voice, television, etc... using virtualized services on a VM machine doesn't seem to work very well either. So if you're using SAS or other cloud based stuff and your users on a virtual machine you run into all sorts of weird transient issues. Lastly, it may seem silly, but if you're company is anything like mine, you have a lot of people that have built their own little tools using Microsoft Access. We didn't realize just how many MS Access tools were out there and how important they were to some departments until we put them on virtual machines and found out that the MS Jet database is a file system based database and definitely does not work well with virtualization... so bad in fact that in many cases it corrupted the database beyond repair. We even tried moving some of the more important ones over to Oracle backends (because frankly, if they're important they should've been all along) but even MS Accesses front end couldn't deal with the virtualization... even though it was an improvement. We finally ended up having all these micro-projects where we had to put all these custom made tools, many of them made by people that didn't know what they were doing and didn't even work for us anymore intro real DBs with real web-based frontends. Reverse engineering some of those was a nightmare. It's a can of worms we likely wouldn't have opened if we had the foresight. I'd personally like to see MS Access treated just like Visual Studio in that only IS is allowed access to it... but I don't make the rules so... whatever.
at my 250k employee company with a bajillion servers and workstations, virtualization is mostly a work-around for the ancient and technophobic company policy of seperating servers by the individual application they run. All of my department's server side stuff (except the database) can easily be run on one box with one active/active failover box in a different location. This is how it was demo'd, benchmarked, vetted, and tested. Corporate audit went ape shit that we had an APPLICATION SERVER that was also SERVING UP WEB CONTENT!!!!!!!!! Fast forward 8 years and we have THIRTY TWO virtual servers in EACH ENVIRONMENT (production and fail-over). All doing the job of one decent server, and realistically all together taking up one server's worth of hardware (but with much more OS overhead BS than just using the one stupid server for the various applications.
So basically we pay some VMWare licenses, 62 extra windows licenses, some hefty maintenence contracts, and who knows what else in order to use the server in the way that the software originally was intended to run (1 box for all the apps). Nice workaround for braindead company policy.
Honestly, other heavily virtualized shops that I have seen were mostly the same thing.
Virtualization in the business world is really just a work around for the fact that computers got more powerful faster than they were able to gain trust and confidence of senior management.
Virtualization is a stone-age technology, useful for crippling hostile environments. This is why "cloud" providers love it, and developers use it for testing. Incompetent sysadmins use it in hope that they can revert their mistakes by switching to pre-fuckup images, having this strange fantasy that this shit is going to fly in a production environment.
If you REALLY need separate hosts for separate applications in production environment (what you almost certainly don't in presence of package manager and usable backup system), there is host partitioning -- VServer, OpenVZ, LXC-based environments, up to schroot-based chroot environments.
Contrary to the popular belief, there indeed is no God.
There are a few "workloads" that just don't like to be virtualized for one reason or another.
Active Directory Domain Controllers -- these work better now under hypervisors, but older versions had a wonderful error message when starting up from a snapshot rollback: "Recovered using an unsupported method", which then resulted in a dead DC that would refuse to do anything until it was restored from a "supported" backup, usually tape. I used to put big "DO NOT SNAPSHOT" warnings on DCs, or even block snapshots with ACLs.
Time-rollback -- Some applications just can't take even a small roll-back in the system clock very well. These have problems when moved from host-to-host with a technology like vMotion. It's usually a coding error, where the system clock is used to "order" transactions, instead of using an incrementing transaction ID counter.
DRM -- lots of apps use hardware-integrated copy protection, like dongles. Some of them can be "made to work" using hardware pass-through, but then you lose the biggest virtualization benefit of being able to migrate VMs around during the day without outages.
Network Latency -- this is a "last but certainly not least". Some badly written applications are very sensitive to network latency because of the use of excessively chatty protocols. Examples are TFTP, reading SQL data row-by-row using cursors, or badly designed RPC protocols that enumerate big lists one item at a time over the wire. Most hypervisors add something like 20-100 microseconds for every packet, and then there are the overheads of context switches, cache thrashing, etc... You can end up with the application performance plummeting, despite faster CPUs and more RAM. I had one case where an application went from taking about an hour to run a report to something like five hours. The arithmetic is simple: Every microsecond of additional latency adds a full second of time per million network round-trips. I've seen applications that do ten billion round-trips to complete a process!
Although most things are no brainers like you mentioned in your post, I'd say my best advice is: Don't guess and do analysis. Clearly you invested into virtualization for a reason and not just because you convinced your boss it was the 'trendy' thing to do. I'm glad to see you're not picking random systems because some poser sys-admin who makes bullshit conjectures said it was a good idea.
On the Windows side, there's plenty of tools to use to gauge virtualization candidacy; I've mostly used perfmon and cross-referenced it with my current performance on VMFS storage luns and what CPU/memory resources I have to spare in what resource pools are available. Linux (which is my primary line of expertise), IMHO, is much easier to gauge. You've got sar, iostat, vmstat (heck the whole sysstat pkg for that matter), not to mention if are a GUI trending fanboi, you've got xymon, nagios, cacti, big brother, etc. to graphically trend performance with nice rrd-tool graphs.
Also, my other factor is your line of business, where your datacenter is and annual budget towards technology update/refresh.
Is the Antichrist. Or the uncle-christ. Or something. Anyway it's damned slow.
Please do not read this sig. Thank you.
It's also very useful for testing, development, prototyping. What better way to have a vast variety of platforms for these tasks?
For prototyping your metal-based services, you can purchase a roomful of servers or you can install vmware. Which is gonna get you to production within your budget?
Unless you write all of your own software, you are going to have old legacy applications in your system, whether you like it or not. If these applications are not CPU intensive but they are still necessary, virtualization is an ideal solution.
I wouldn't virtualize our delivery truck. That's be bad.
Oh, and our stores' inventory. That'd be bad virtualized, too.
I don't respond to AC's.
This is FUD and misinformation. Most things are great until is breaks. What is your point?
Whether or not you virtualize a given workload, you need to manage its resource usage. In the trivial case, you can see your CPU used up by some other program unless you provide your program a guarantee of enough CPU for the actual number of users who will be employing it.
Ditto memory, disk and networ I/O, bus bandwidth, etc, etc.
A more surprising case is putting two workloads together that formerly worked properly on an older, slower machine. If you increase the amount of a critical resource, both programs under load will start using more of every resource. For example, a batch job that got 30% more CPU increased the amount of disk I/O it did by several times. An interactive program on the same machine was rendered almost completely unusable because it couldn't do the I/O it needed. The customer in that case thought the vendor was lying about the speed of the machine, and demanded his old one back.
Linux is a hotbed of resource management experimentation, so you can statically size and configure a program (workload) to be able to withstand a given load. Commercial Unixes have good enough controls to do most common cases. I can't speak about Windows and BSD, as I've not researched them (yet). Mainframes, not surprisingly, have the best controls for what in their days were exceedingly precious resources.
If your OS doesn't have good resource controls, or if you don't know how to use them well, you'll end up splitting up the virtual machines onto a undesirably large number of physical machines, just in order to do the management the hard way.
The difficulty, by the way, varies as something like the square of the number of machines and the resources used, so virtualization and consolidation is easy for well-behaved, small and unimportant programs, and can be evil for anything that turns out to be big, resource-intensive or important. Think of that as a lemma used to derive Murphy's law (:-)).
--dave
davecb@spamcop.net
Comment removed based on user account deletion
I had a job where you would have to connect to a virtual desktop for actual work. The local operating system of the computer you were actually using was only for connecting to the virtual one. It was a horrible experience and very frustrating. Please never force others to work like that.
Isn't it funny how two people can look at the exact same computing setup...
One person says "hardware is cheap so we give the users maximal control"
The other says "what a terrible waste of resources"
The hardware costs you speak of are ZIP ZERO NADA compared to potential profits.
I wouldn't virtualize a firewall.
-Z
What possible reason is there for doing this??? Oracle spends much time and effort on their product's features, just so you don't have to resort to this sort of thing.
There are plenty of valid reasons for virtualization but running a database server is not one of them, esp. if you are complaining about performance.
Specifically, I hear that it's generally not a good idea to virtualize Asterisk servers that you expect will be handling more than 12-15 simultaneous connections. However, at the moment I've got three Asterisk servers that are shouldering lighter loads and running just fine on qemu-kvm virtual machines.
It will suffer specifically when you put multiple number-crunching VMs on the same physical host, not specifically because there is a "virtualization overhead", but because when you try to consolidate workloads with heavy demand for a limited resource, there will be contention; on the dedicated host it doesn't have to share, so there will be more CPU time available that is not scheduled for other number-crunchers.
You can avoid the problem with proper capacity planning
That including VoIP, video-conferencing, online game servers and anything realtime.
If such services exist, the firewall too should not be virtualized.
"Virtualization" always means something that emulates a host hardware that runs a separate and full operating system instance. Most such environments include "acceleration" that bypasses emulation for "important" functionality, but the defining characteristic is the existence of fake hardware and full OS running on it. It has no place in production environment unless Windows is involved.
Host partitioning is a completely different technology, it keeps single instance of OS kernel (or at least large part of it) and provides multiple interfaces that from userspace look like isolated environments. It's a legitimate technology, and it should not be lumped under "virtualization" umbrella.
Contrary to the popular belief, there indeed is no God.
Including the servers. No need for any hardware at all. It's VMs all the way down.
Or maybe you could just virtualize the CIO.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
It's all work that needs to be done anyway.
"people who run Windows do not care about performance. Or security. Or reliability. "
Yeah they might be FACTORY WORKERS or DOCTORS or LAWYERS running an INDUSTRY-SPECIFIC APPLICATION.
You know, the people who actually EARN the money that pays IT's salary.
The whole article is worded as though written by an advertiser. This is nothing but Slashdot Market Research. Either it will be a hit business article, "What Not to Try and Virtualize, Straight from the Engineers" or research into how segments of the industry can convince you to virtualize that anyway.
Must be nice, buy one website and you end up with a corralled group of wise and experienced IT gurus. Then slaughter them like sheep. This post was nothing but Market Research. Move along.
You clearly have no idea what you are talking about if you think things like LPARs have no place in a production environment.
You don't need all your LDAP and/or other Auth, DNS, and other core services on physical hardware, but make sure that if your whole virt environment goes down you can get it back up again...
This is why there are cross compilers
Virtualization has specific use cases that are interesting. Compile farms are not one of them
Wow, you seem to think that virtual machines have some sort of magic "segregation" that prevents them from hacking their way into the host kernel!
Are you willing to assert that there are no buffer overun bugs in vmware's drivers? Unless you can make that assertion, then you are wildly off-base with your security assumptions.
I see loads of comments that I agree with about vm best practices, but I'll add this generalization:
It sounds like you have a small-ish setup. Virtualization is a little different when you get below a certain size, because you don't have the budget to throw at it. My last project took some 150 servers into 15 physical machines. Lots and lots of purpose-specific servers running some little app or service alongside the normal big servers handling file serving, print serving and enterprise apps. The 15 machines were beefy, but they were the cheap part. The storage was the thing. We blew over a quarter million on a nice SAN to go with the two we already had. With high-end storage you can do just about anything you want with virtualization. In fact, lots of things will suddenly perform much better due to the time-sharing of better hardware. Well, and the network. It turns out you need a lot of fast and well managed ports to run a VMware farm.
We got huge benefits from virtualizing DEV, QA and Staging environments. They were always way, way underpowered and cramped before virtualization. Also, we were able to rip off copies of anything we needed to experiment with in seconds. Huge, huge benefit there. And not just from the server virtualization - there's a lot of network hardware that we never would have been able to afford that just gets virtualized. In DEV and QA we routinely ripped off copies of a whole suite of related servers and stuck them in their own private network just to try something out - something you could never do with physical hardware.
Anyway, if you can afford the hardware you need for servers and storage and network - virtualize everything that doesn't have a good reason not to be virtualized. Ideally set up server farms in two locations and use SAN and VMware features to keep instant-on copies of everything in two locations. It takes a bit of work and more than a tiny bit of cash, but you'll be bulletproof when you are finished. Actually, I don't want to underplay the work involved in full redundancy - it isn't just VMware, there's work to be done at every level, particularly at the network level. But once it is set up correctly, your workload drops precipitously. A VM farm will be much, much less work to maintain than the physical servers - and your users will perceive a noticeable uptick in reliability. With a correctly sized and architected farm you'll never have downtime due to hardware problems again. This is a powerful argument for forcing everything you can possibly justify into virtualization.
We had a few reluctant business units and a couple of vendors that wouldn't certify their stuff on VMware (this was 2005-2008 era), so we kept the old hardware server updated and ready to go as a bailout plan and virtualized them one at a time. Never had a problem, everything worked better on VMware. The desktop project on the other hand... well, don't get me started...
So in other words, if the use case falls within the parameters you've set out, and you can manage to get some help from someone who knows what they are doing, then virtualization can be a big win for you.
And then you push HARD on the straw man: "the biggest virtualization benefit of being able to migrate VMs around during the day without outages."
but there are PLENTY of other reasons that are perfectly valid! There are plenty of good use cases for virtual machines where it's just fine if the virtual machine stays right where it is.
But don't let me stop you from making overblown, over-generalized pronouncements
I know.. I sound like a open source geek.. but really.. check it.
For instance, you can run Oracle Standard Edition on up to 2 physical CPU's. However, according to Oracle, you must count EACH physical CPU in the server its running on (unless, of course, you are using Oracle VM, which is the only exception they allow, I know.. surprising).
If you put Your oracle DB on a VMWare server running with 4 CPUs'... congratulations, you now need to upgrade to enterprise edition. (and easily at least one more zero in the price, since now you also have to license every core, and feature)
What are we going to do tonight Brain?
Remote sysloggers :)
Firewalls
Visualisation hosts
Virtualization is one of those things that is a great tool, but people assume it is an all or nothing item, mainly because there are so many vendors making cash on VM solutions, as well as vendors making cash on discrete hardware solutions.
Because it is a grey area, usually a "real" solution doesn't happen with all the extremely loud voices beating the drumbeats for their technologies.
There are things that I just wouldn't bother virtualizing. Anything that has to work near RT for example like NTP servers. I also like keeping very sensitive items on physically separate frames. I do trust hypervisor security (it is good enough for production databases), but there are some items where I like it not in the mix, such as Netbackup master/media servers, LDAP servers, SecurID boxes, SDMC boxes, and the syslog dump boxes. Having LDAP on its own discrete frame also means that even if something takes down the VMWare cluster, people are still able to authenticate and do something. Same with SecurID. Someone knocks that out, there goes access from outside the perimeter, and that can cause some screaming, and may even cause a chicken-and-egg scenario if someone configures SecurID on vCenter.
The trick is to use virtualization and the cloud as tools. However, so many people have their interests lie in one blind direction or the other that finding a solution that uses the best tool for the job is difficult.
LPAR is an IBM term, and it's only applicable to their platform, a legacy of what I have already describes as stone age of computing. Generic non-virtualization mechanism for achieving the same is hardware partitioning, and it works great with operating systems that were developed as an alternative to ancient IBM architecture.
Contrary to the popular belief, there indeed is no God.
'cause if you knock it offline by accident, your easiest tool with which to bring it back online is gone?
Ditto with the online manuals for the virtualizer.
This problem was described in 1961, in a short story by Hal Draper titled: MS Fnd in a Lbry.
Kind of like how it's a bad idea to mess with a host's eth0 settings if you're currently logged in via ssh through eth0.
Or putting a breakpoint in the debugger's breakpoint handling routine. Or single-stepping into the debugger.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
s/hardware partitioning/host partitioning/
Contrary to the popular belief, there indeed is no God.
By the way: This is one of the things that's a problem with systems (like lisp and smalltalk) where the development environment is part of the program under development, rather than a separate toolset which treats the target code as pure data.
Modifying the environment itself can be a major hassle. And it breaks isolation when developing something intended to stand alone, dragging in a lot of unrelated code.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
We run a 1200 core HPC cluster with 900TB of storage. Our workflows are mostly CERN analysis jobs that come through Grid so we don't really have a control on what exactly the user is doing in his/her job. We used to allow up to the number of cores of jobs in any worknode, but had issues with misbehaving jobs sometimes killing off the whole node taking the other jobs with it when some code ran amok on memory allocations or what not. We attempted various resource controls, but still at times they managed to make the node unresponsive.
Since about 1 month we run 100% virtualized OpenVZ cluster with every compute core as a VM (and we actually over-provision by 10% as a lot of jobs are data intense and not CPU intense). So far no single node has died due to bad user behavior. Single VM's have ran into OOM, but as they are virtual containers the jobs are killed inside and the VM recovers nicely allowing the next job to execute. As far as we've seen studies and tested the OpenVZ container based virtualization doesn't have the drawbacks of local disk I/O and network overheads and as it's just a container the overhead on CPU is negligible. Also what it allows us is that if the experiments declare a new needed workernode software content we can just deploy one such instance, clone it and redeploy all VM's far faster than we could reinstall the nodes. (The HN's run CentOS 6 while the WN's require SL5 for example so as long as you remain with Linux you're fine).
We also run our central scheduling servers and Grid CE's and information systems and what not in the same OpenVZ containers allowing for example to live migrate between nodes to allow for service work and allowing 24 core / 128GB ram nodes to be used for 30-40 services while officially those services require exclusive OS instances because of conflicting binary and library versions. Since we went virtual we've risen in availability ratings and the management overhead has significantly dropped. We're going to expand the cluster in the next 1-2 months to 5900 job slots and 2PB of storage running on 10GbE non-blocking interconnect and expect the whole thing to be a real nice boost in our computing capacity and availability.
Of course our model relies on the fact that CERN doesn't use multi core jobs, but instead splits the computational tasks into subtasks each working quite nicely on a single core therefore allowing such virtualization to work perfectly. For MPI applications one would just have to use different kind of VM configs and possibly auto-deploy based on job requirements, but I don't see that you'd lose much unless your each job requires a full hardnode or even multiple of them. And actually even in those cases running one VM per HN might not be a bad idea for software maintenance as well as starting from a fresh clean VM every time a job starts if you really want to (we've not done an auto-deploy system as our jobs all use the same requirements, but it's not hard considering a VM deploy on OpenVZ usually takes less than a minute from image to running VM).
Anything realtime.
We deliver realtime control systems and two things we never do is virtualization and 'puting it in the cloud'. We need the PLC to have an answer within 100msecs or capacity will go down.
-- The Internet is a too slow way of doing things, you'd never do without it.
If the machine to virtualize on is more expensive than than the hardware things are coming from, you might not weant to virtualise. There is a little overhead, most important for I/O and bandwidth, so these you should look at.
Our experience is, that in many cases SAN allocation is handled with a forklift, giving you first available space, instead of analysing disk load, and allocate spindles to the purpose. So SAN often will give a negative impact to your performance due to the SAN people not being up to the job.
If you understand this question:
"PCIE SSDs are advertised to deliver 100,000 to 500,000 IOPs. Has anyone experimented with PCI-Express based SSD solutions in their VM hosts to keep high IO VMs like VDI and SQL from swamping their SAN infrastructure?"
you are a computer expert.
We suffer more in our imagination than in reality. - Seneca
A little late to the party, but I'd say backups.
You want access to your backups, no matter what the state of your virtualization infrastructure.
But I would hesitate to run more than one VM per physical machine in most cases. And in any case, I have vmware PLAYER blow up on me and cause problems all the time, I don't really trust vmware to run a bunch of machines on one host. I've known too many people who have worked for vmware.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
We'll never virtualize our BOFH.
cpghost at Cordula's Web.
A current topic of debate at work is whether or not Virtualized Machines (VM) can be reliably used for performance testing. Specifically, can VMs be used to host LoadRunner test software to generate load against the targeted test system? The current answer seems to be that virtualization is an acceptable host in a single tenant model where the host server is dedicated to performance testing and is not over-subscribed.
Nuff said.
Lisias@Earth.SolarSystem.OrionArm.MilkyWay.Local.Virgo.Universe.org
But how about you know what you're doing ? Failing that, make sure you have 2 hosts in the same vlan. Fucking up eth0 beyond the point where you can, if you know what you're doing, connect from a host in the same LAN is quite hard.
...I'll let you figure out why, suffice to say that I've seen someone try it, fully believing they were somehow gaining CPU cycles... oh how we laughed!
You can virtualize a lot of stuff in the data center. Some of it brings high return on investment and others don't. You want to do a needs analysis on existing infrastructure and expected resource level commitments over the medium and long term.
Then what you'll find is that you should definitely not virtualize your underpants. Underpants should definitely stay in the realm of the non-virtual, and preferably right there on your asset.
The rest of the stuff? Meh.
If this were Usenet, I'd killfile the lot of you.
Anything that requires a lot of dedicated RAM and fills it's allotment up regardless (such as databases and good file servers will cache).
I actually don't virtualize anything server-side because where I work is small enough that it will actually cost more. We do use chroot'ed environments for DNS, Apache etc but they all run on the same host, there is actually little that virtualizing would do for us (besides giving me more machines to patch/configure).
What is good to virtualize is any Windows environment, I give my clients their Windows environments virtually, it saves me a lot of time and they can always jump back easily if they screw up. The only things we need on Windows are very specific scripts and software that's only used once in a while.
Custom electronics and digital signage for your business: www.evcircuits.com
If you're looking for a performance hike by binding a thread to a CPU using taskset or ThreadAffinity, you'll want to steer well clear of any kind of VM. In fact, if you're seeking any performance benefit by controlling what happens on the metal, VM is going to make that impossible.
The latest offering from IBM scales to 300 virtualized servers (on a z114 in the cheapest config, realisticly @ $100K). Redundancy is built in the hardware. CRC is done at the hardware level. Mind you, these systems are designed to run at 100% utilization non-stop. Not this if higher than 15% you should run physical c(r)ap from VMware.
The best weapon of a dictatorship is secrecy, but the best weapon of a democracy should be the weapon of openness.
Ah, yes, 'the stone age of computing'. Everything that old MUST be worse than new stuff, right? Let's do a little comparison, shall we (remember, you are the one who made the statement 'achieve the same thing')?
What, exactly, can you do with it? LPAR - run any OS (z/VM, z/OS, z/VSE, Linux, OpenSolaris) that can run on z/Architecture. VServer - run Linux, and only Linux.
What are the security statements for each? LPAR - certified EAL5, worldwide. VServer - 'we hope it's secure' (from the Vserver FAQ).
How isolated are the guests from each other and the host? LPAR - 100% isolation, see EAL5 statement. VServer - FAQ has all sorts of things like a guest being able to take down not only it's own network, but the networks of the host and other guests. Also has tips on how to configure things so guests and the host don't clash. In other words, not very isolated at all.
Is there any difference between running native and running as a guest? LPAR - no. VServer - FAQ has various tips on configuring programs, as well as a list of 'problematic programs', so yes, there is a difference.
There are many other differences also. Not one of those differences provides any sort of advantage to host partitioning.
If LPAR and its' ilk are 'stone age', host partitioning is mired firmly in the Jurassic.
Ive seen a couple companies have a problem where the whole underlying fabric of their VM architecture render them broken for almost a week. During this time none of their VMs were accessible. My advise is if there is one thing your company just cant live without, dont put that in your VM cloud. One example would be PBX software. If you live and die by your phones, then having them on a separate independent system with its own built in redundancies will do more to ensure some level of survivability during a disaster.
Anything that requires promiscuous mode or low level access to the network drivers. The extra layer of virtualization and the generic VM network drivers prevents this type of access.
Anything that would take up all of the resources for any of the key system components (i.e. CPU, Disk I/O, Network I/O). Some examples of these would usually be Database servers, Backup master servers, etc.
Everything that old MUST be worse than new stuff, right?
No. However the fact that everyone except IBM rejected their architectural decisions, suggests that those decisions are pretty bad.
What are the security statements for each? LPAR - certified EAL5, worldwide.
Wake me up when "certifications" are revoked at the first update to the "certified" system or first security bug found in it.
VServer - 'we hope it's secure' (from the Vserver FAQ).
That's a great metric of system security -- amount of boasting the developers produce about it.
Contrary to the popular belief, there indeed is no God.
Ask Slashdot: What Type of Asset Would You Not Virtualize?
^^^ That, I wouldn't virtualize (or at least I would think over it very seriously.)
Uh, yeah, z/Architecture is 'pretty bad'. I mean, it's not like it has continued evolving for almost 50 years, while at the same time never requiring a single user to rewrite, port, or even recompile any application. Try not to be so stupid. Nobody 'rejected' IBMs architecture, they just find it cheaper and easier to buy someone else's chip instead of designing their own around zArchitecture (IBM does not sell those chips to anyone else). Take a look at any modern computer, remove all of IBMs 'bad architecture' decisions from it, and see what you have left. Start with the idea of an 'instruction set architecture' that stays constant across models and generations. Then move on to the architecture of I/O being independent of the processor, so that the same I/O devices can be used on different models and generations. Oh, and let's not forget other minor details like cache. And 8-bit bytes. And 32-bit words. And byte-addressable memory. And DAT. Yep, those minor architecture issues sure were rejected by everyone except IBM.
EAL5 does not say or imply that there are no bugs. EAL5 says that specific international standards were followed in the design and implementation. Suse and Red Hat enterprise Linux have been certified at EAL4, as has Windows 7 and Windows XP.
No, what a developer says is not a great metric of system security. Much better to have an independent metric of security (such as EAL). However, when the developers 'we hope it is secure' statement is the ONLY metric of security assurance you have, there is nothing else to use.
Nobody 'rejected' IBMs architecture, they just find it cheaper and easier to buy someone else's chip instead of designing their own around zArchitecture (IBM does not sell those chips to anyone else).
"Someone else" (all of them) chosen a simpler CPU architecture for a reason.
EAL5 does not say or imply that there are no bugs. EAL5 says that specific international standards were followed in the design and implementation. Suse and Red Hat enterprise Linux have been certified at EAL4, as has Windows 7 and Windows XP.
And as long as there are full-compromise bugs slipping through certifications, those "standards" are meaningless.
Contrary to the popular belief, there indeed is no God.
this is an excellent breakdown.
DRM: Terminator crops for your mind!
Yeah, I told you why they (computer manufacturers) picked other CPUs - so they don't have to develop their own. If you have on one hand an architecture for which someone will sell you a processor, and on the other hand an architecture (no matter the merits of that architecture) where you have to develop your OWN processor (and probably pay royalties on it anyway), which are you going to pick? That has absolutely NOTHING to do with the merits of the architecture.
On the other hand, if you are a processor manufacturer, which would you rather do? Implement someone else's instruction set, so they get to set your direction, or make up your own, so you get to set your own direction? Of course, while designing your architecture you would probably look at other architectures and discard all the stuff from 'bad' and 'expensive' architectures. Which is why today almost all processors are 6-bit-byte machines with 15 bit words (word addressable only), the instruction set changes depending on the performance of the processor, the I/O also changes with each processor iteration, have no cache, have no concept of supervisor/problem states, and their 'virtual memory' involves swapping out/in ALL of real memory on every task switch (due to the lack of DAT and all the complexities that adds).
Oh, and in case you have the mistaken belief that a complex instruction set means a more complicated chip, you are wrong there also. All of the complex stuff is done in microcode, another 'bad' architecture from IBM.
Please provide a list of 'full compromises' involving systems certified at EAL5 or higher.
Who, do you think, invented instruction sets and memory models?
Speaking of which, even Intel had to fundamentally re-do its memory model -- in fact, three times in the history of x86 series.
Contrary to the popular belief, there indeed is no God.
We have a build farm using both optimized NFS and SAN-base filesystems. The SAN is a lot faster than the NFS but also far more expensive. However, local disk is far faster than both of them but makes it hard to do virtual machine failover.
I would dearly love to have a physical cluster of machines for building on (a single build takes roughly 20 hours on our build farm) but the IT gods have decreed that the build farm SHALL be virtualized.
There are always compromises. You just have to figure out what is most important to you. If your goal is 100% virtualization, then you are going to have to be more diligent about designing, planning, and maintaining the environment. If you are not the only person who will be responsible for the environment, you have to make sure that you put in a solution that whoever else is touching the environment can support as well - one well-meaning, but improperly-informed person can do a lot of damage in even the best-planned environment.
Personally, I would always leave at least one physical server for AD/DNS (which I would give the PDC Emulator role, since it's a fairly resource intensive one and you're likely to have plenty of spare cycles on the physical server).
As long as your environment isn't especially complex, you will probably be better off virtualizing vCenter. However, depending on your environment, there may be justification for keeping vCenter physical.
I like to have some kind of monitoring platform that is physical - it's never good to have a catastrophic failure that you don't get notified of because your monitoring software was one of the affected VMs, or your mail server VM was down and the alerts didn't make it to you (or you can virtualize your monitoring platform, and have something physical that monitors your monitoring platform - possibly something with a SMS gateway, which frequently only seem to work right with physical servers anyways). Generally, you can put this on either your physical AD/DNS server, or you vCenter server if you have one.
Any server that relies on any specific hardware to operate is certainly not a virtualization candidate, unless the vendor has a solution. Fax servers (typically requiring fax boards), servers with hardware dongles for licensing (although, sometimes you can use USB or Serial over IP devices to work around this limitation), etc.
Any highly performant database server (especially if clustered) deserves a second look before virtualizing. Not that it can't be done, but there can be an awful lot of suffering if you don't get it right the first time.
Any software vendor with whom you have support contracts should be consulted before virtualizing the corresponding server - some simply refuse to support their products when virtualized. Sometimes you can lie about it and still get support, but some of them will remote in and check for VMware services, processes, etc and not provide support if they are found.
It sounds to me that the "Use" of the Exchange infrastructure in question may have grown larger than the infrastructure could, which yes, would cause issues, physical or virtual. Or perhaps the SAN environment being used wasn't very "friendly" to Exchange.
Agreed. If virtualizing Exchange broke the environment, then the environment wasn't sized properly. Too few servers, not enough IOPS, something. Of all things, Exchange is the one thing that I would almost never consider running physical. 2003, 2007, 2010 - all work flawlessly virtualized. The only thing I've not tested is the Unified Messaging role, which until recently wasn't supported virtualized.
Anything with high system requirements is not a great candidate for virtualization. One of the major selling points is taking ten servers that average 5% CPU and combining them onto a single host to realize cost savings over having to replace those 10 physical servers at some point.
If you have a quad cpu db server then you're likely going to have to dedicate a host to run that single VM, so you're still paying the same hardware costs, adding in VM software costs, and slightly reducing the performance by introducing the overhead of a hypervisor.
Other problems are systems that have specialized hardware, i.e. something that you have to connect to a serial port. That doesn't work well in a virtualized environment.
My preference is always to keep the system running backups on a separate physical server. Reason for this is 1) the backup server should not be so critical as to impact your production apps if there is a problem with it, 2) restoring from a VM catastrophe is a lot easier if you don't need the VM infrastructure to work to perform a restore, 3) it's a lot easier to throw a server and a tape drive in your trunk if you need to run to escape some sort of disaster than it is to throw multiple VM hosts, SAN, etc in there, and 4) I prefer not to share bus bandwidths with running VMs when running backups.