Massive VMware Bug Shuts Systems Down

mattmarlowe writes "Imagine if Red Hat released a version of Linux, and after it was deployed, customers noticed that any processes with a start date of today would refuse to run? Well, that's what happened to VMware — a company that wants nearly all server applications running in virtual machines within a matter of years." Supposedly a fix will be available ... in 36 hours.

Comments

License Management Software!?

[bigtallmofo]

I don't get license management measures in software that is only going to be used by major corporations.

If someone wants to run virtual machines at home or in a small business, they're likely going to be more than satisfied with VMWare Virtual Server (formerly GSX) and wouldn't even consider the much more complex ESX.

In a major corporation, fear of massive fines and prosecution is enough to stop them from pirating your software. Hardware dongles, software license managers and the like only hurt your paying customers.

Re:License Management Software!?

[db32]

Exactly. It is a tremendous pain in the ass to track all the stupid license keys and crap in use. Departments frequently need software specific to only their department and outside the scope of normal IT support stuff. Phone numbers, licenses, etc. God forbid any of those companies get purchased or go under, then you are stuck with expensive software that you cannot recover.

The call home variety is extremely infuriating. On top of whatever nonsense key/activation crap you have to go through, you have to put up with it trying to call home or deactivating itself. MS isn't the only guilty party in this, but those bastards certainly made the situation much worse.

Re:License Management Software!?

[morgan_greywolf]

Exactly. Most large companies usually have an entire person, and sometimes multiple people dedicated to nothing but license management.

What a colossal waste of money.

Re:License Management Software!?

[shawn(at)fsu]

I really don't think the fines would keep large corporations in line. look at all the stuff you see big business doing that they know is illegal and that they know will land them big fines if they get caught. Software piracy is no different. In fact it's probably easier to use a pirated piece of software than it is to dump illegal chemicals or defraud investors. You can manage the exposer.

Re:License Management Software!?

[rudeboy1]

Good god do I hear you, brother. I work IT for a legal firm. So many little apps no one else in IT has ever even heard of. And most of them, you're talking to the same guy for support that developed it, and filled the sales order. Out of his basement or garage. Multi-million dollar a year law firm, and it can be brought to its knees if one of our obscure applications goes down and needs support, and the one guy that can support it is out taking his kids to soccer practice.

I'm looking at you North Winds Software. I'll BUY a support contract! If you offered such a thing. If you answered the phone.

I need to go back to bed. :(

Re:License Management Software!?

[_merlin]

Having administered ESX, I can say the license management is useful for one thing: it helps you ensure you aren't exceeding what you're licensed for. For example, if you aren't licensed for multi-processor boxes, it will complain until you get a valid license. If nothing else, it gives you some confidence that you will pass an audit.

License management is also useful for things like MATLAB and OPNET that are licensed per concurrent user: you can install on as many machines as you like, but they need to be able to talk to your license server (not that this is _your_ license server on your network - it isn't "calling home") to ensure that the number of concurrent users is below the maximum allowed. That way, if say, everyone needs to be able to run OPNET occasionally, but not very often, everyone can install it, but you only need to pay for a few licenses. You know you aren't exceeding your licenses because it won't let you launch more instances than you're allowed simultaneously. If your users regularly complain that they can't fire up OPNET due to lack of licenses, you pay for a few more seats.

On the other hand, I can't stand software that calls home to ensure that it's "genuine" a la Windows Vista, or those stupid CD copy protection schemes. That's bullshit. Things like that make more work for a sysadmin, not less. I only like license management when it helps me, the admin; I don't care what it does or doesn't do for the software vendor. I'm a selfish pig, I know.

Another thing I can't stand is things like Rational Purify where they attempt to count your "activations" at their end: when you install Purify, it increases the installed count in IBM's system, and decreases it when you uninstall. If the IBM server thinks you're using all your licenses, you can't install. Too bad people always forget to uninstall Purify before wiping their computers for a clean OS install (or scrapping the computers)! And don't get me started on how bad it is to deal with IBM's phone support. This is one copy protection scheme that I do bypass: I install Purify in a VMware virtual machine, snapshot it, uninstall Purify, and roll the virtual machine back to the snapshot. That way, Purify will work in the virtual machine, but IBM's servers will think I haven't used any of my licenses. Also, I can make copies of the virtual machine for multiple people to use. It's easier for me to track the licences than put up with a crap license management scheme.

Re:License Management Software!?

[supersnail]

Actually its quite a common policy in MegaCorps to reject software that require machine specific or expiring license keys for use in "Mission Critical" applications.

The backup server not having the correct licenses is one of the biggest risks in a Disaster Recovery.

Migration to newer better hardware also becomes a nightmare where license keys are involved -- what do you mean the new server doesnt have centronics port for the dongle?

Its also screws up the companys virtualisation strategy as you have no idea whether a given license scheme will work in inside a VM or not.

Do like the Fortune 500 and just say no to runtime licenses.
       

Re:License Management Software!?

[Gordonjcp]

I'm looking at you North Winds Software. I'll BUY a support contract! If you offered such a thing. If you answered the phone.

There's an Ask Slashdot for you. Is there something out there that can replace this magic bit of software? Is anyone interested in writing an Open-Source equivalent?

Re:License Management Software!?

[db32]

There is a WIDE WIDE range of things that don't exist in the F/OSS world yet. The killer problem seems to be inherent in the way F/OSS works. Industry specific things frequently don't happen unless people from that industry also happen to be coders. Outside of the inherent difficulty in writing software for an industry you don't understand, most geeks don't bother to learn about other industries and instead assume that they should all operate the same way IT does.

Re:License Management Software!?

[drachenstern]

The more important variant of that question is does the parent want to share enough of the details of operation (clean room style) to get someone to want to write an OS equiv.

Don't misunderstand me, I like to write code, but if I don't know what the hole looks like, I can't carve a peg to fit it...

Re:License Management Software!?

[swabeui]

I'm looking at you North Winds Software. I'll BUY a support contract! If you offered such a thing. If you answered the phone.

North Winds Software? Just a WILD guess... is this 'software' based on MS Access? I wonder where they got the company name from...

Re:License Management Software!?

[fr175]

I really don't think the fines would keep large corporations in line. look at all the stuff you see big business doing that they know is illegal and that they know will land them big fines if they get caught. Software piracy is no different. In fact it's probably easier to use a pirated piece of software than it is to dump illegal chemicals or defraud investors. You can manage the exposer.

Having acted in an advising capacity on a software license management project currently underway at one of the worlds largest financial institutions (400k employees), I disagree. Purchased software is an asset on the books and needs to be tracked. Pirated software is a risk and even the largest companies will occasionally be brought to court for "over implementation."

The main hurdle with Software Asset Management (SAM) is the complexity of the licenses involved, and the multitude of way in which it can be obtained. Some examples: is the license perpetual or subscription based; is it a "named user" license or is it assigned to the org; does it include maintenance (upgrade rights); if it includes maintenance is the maint co-termed with the other licenses that the org owns; if it includes maintenance, what was the most current version at the time the maintenance expired; does the current version allow for "downgrades" and how many version prior can be downgraded; what previous versions qualify for an upgrade license and which would need a full new version; can the licenses be transferred within the org; can they be transferred globally; does the license allow for home use; does the license allow for portable device use; just to name a few.

If large corporations were willing pirates, you would not see them making their annual multi-million dollar payments to Microsoft for their Enterprise Agreements. You wouldn't see them spending millions on risk management/mitigation consultants or conducting their own software audits. There are people out there getting paid piles of cash to implement a working SAM system.

It's unavoidable that a large corporation will be under-licensed. However, they spend big bucks to mitigate the risk that this opens them up to.

Re:License Management Software!?

[alohatiger]

What is a "disadvantaged business" anyway, and why would someone actually use that as a sales point?

Government work. Some government contracts require a percentage of the work to be done by minority/women/veteran/disadvantaged owned businesses.

Re:License Management Software!?

[Skjellifetti]

In a major corporation, fear of massive fines and prosecution is enough to stop them from pirating your software.

Not true. I worked for a smallish software companies that had their software replicated in at least one large customer installation well beyond the number of seats that were actually paid for. When confronted, the reaction was "so sue us..." We eventually settled for about 1/10 of what we would have made if they had obeyed the license terms because the cost of litigation coupled with the delay tactics they could have used would have meant that we would be out of business long before the court case was over. Size just means that they have more resources to defend their slimy actions.

Re:License Management Software!?

[tsstahl]

I'm looking at you North Winds Software. I'll BUY a support contract! If you offered such a thing. If you answered the phone.

Um, isn't North Winds the name of the company that comes with the sample Access database? They're not real, you know... ;)

Re:License Management Software!?

[MBGMorden]

and as far as everyone else in the business is concerned, any failings in the product is the IT department's problem not theirs

This is true, and particularly frustrating. We recently have converted from an (old, but very functional and stable) 20+ year old COBOL program to a new Windows application in our organization. This is a Visual Basic application that if I'm being kind I'd say is a kludge held together by the electronic equivalent of duct tape and glue. The thing is junk and crashes ALL THE TIME. IT didn't pick this app though - we just get stuck supporting it. However, no amount of explanation can convince these people that the program crashing is not IT's fault. We can reinstall it as many times as they ask for it. We can update everything on their computer. We can buy them a new computer. But the basic fact is the program you bought is crap and full of bugs and nothing IT does is going to make it stop crashing and screwing up data.

Sadly, this is a hard fact to make users accept.

Re:License Management Software!?

[jgarra23]

Government work. Some government contracts require a percentage of the work to be done by minority/women/veteran/disadvantaged owned businesses.

There aren't enough minority/women/veteran/disadvantaged heart surgeons out there! I demand that we establish hiring quotas for all! There needs to be ethnic & cultural diversity within the heart surgeon community!! :)

Re:License Management Software!?

[Anonymous Coward]

Free software can get companies shut down, and corporate officers put in prison, due to Sarbanes-Oxley or HIPAA law violations.

One example. I have two operating systems. One is a F/OSS distribution of Linux. The other is Windows or a FIPS certified Linux distribution like SUSE or RedHat. This is a company that is publically traded, so falls under Sarbox.

Someone penetrates the machine via a bug in the OS and causes damage, or obtains info. With Windows, or a certified OS, I can tell the auditors that the operating system was certified by the US government with FIPS and/or Common Criteria, and I have used due diligence.

With a non certified OS, I do not have this protection. I can be held culpable because I did not follow due diligence. If the shareholders are pissed enough, someone is going to prison, and the company may be shut down by the SEC if the breach was big enough.

Free software is unusable by a lot of businesses for this, and contract reasons. I know a lot of businesses who pay Symantec and Mcafee tens of thousands of dollars a year for antivirus products installed on Solaris and AIX machines. Not because they will ever see a virus, but to check off a contract clause.

Re:License Management Software!?

[initdeep]

you've never been to a heart surgeon have you?

Many of them are minorities.

in fact, in my experience (having a parent formerly work for one, and the other now seeing one regularly), the people considered minorities in society make up the majority of the heart surgeons.

Re:License Management Software!?

[Anonymous Coward]

I'm on the other side of the divide---a tiny company that's not too much more than a guy in his garage (just a few of us), and frankly, I agree with you. I'm astonished at the way we do things, even though we sell to huge firms (including big law firms, like yours). Part of it is just size---we don't have the people or skills to do all the safety, security and support steps a big corporation would. Still, freaks me out that the crap I wrote is out there being used to do important things by important people who don't realize how dumb the guy who wrote their software actually is.

Re:License Management Software!?

[IntlHarvester]

The problem is that every industry has a few of these super-specialized vertical apps that come from one-guy software companies. Most of them are fairly simplistic Access/FoxPro type things, the hard part was implementing all of the business rules.

I've worked with a few companies that recreated their software package in-house (because they needed specific customization the author wouldn't provide), and it's never as cheap or easy as it might seem superficially.

Re:License Management Software!?

[lukas84]

Add to the fact that this sort of business software is usually extremely boring to write and develop, and requires much more process knowledge than programming knowledge.

Re:License Management Software!?

[SpaceLifeForm]

Well, that explains the lack of support.

Can't start processes?

[oldspewey]

any processes with a start date of today would refuse to run? Supposedly a fix will be available... in 36 hours.

Good thing the fix will be available tomorrow, because if it was available today nobody would be able to run the update process

what do you expect?

[larry bagina]

Who knows what else is lurking in their code base? Certainly not me or you -- we can't see it. We're at their mercy to find and fix problems.

I stick to virtualbox. I'm not going to pretend I've audited the source code, but if I need to, I can.

Say YES to freedom.

Re:what do you expect?

[Anonymous Coward]

Then give me USB support in VirtualBox. Cause I kinda need that the most.

Re:what do you expect?

[dctoastman]

http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf

What if you can't even trust your compiler? At some point, even with fully open, GPL-compliant software, there is some point you just have to trust someone else to not jack you.

Re:what do you expect?

[tlacuache]

I've heard people say this, and I'm honestly curious... what exactly do you need USB for in your virtual machines? Printing? Webcam? I use VirtualBox basically so I can run a few Windows-only apps. For copying files between the host and the guest I use SCP. I print over the network. I'm not trolling, I'm honestly curious. What USB hardware do you need in your VMs?

Re:what do you expect?

[ray-auch]

USB license dongle for the application software running on the VM.

Seriously. Last week.

Re:what do you expect?

[John Hasler]

> What if you can't even trust your compiler?

You are referring to "Reflections on Trusting Trust" I assume. That is not really a practical attack in the real world.

> At some point, even with fully open, GPL-compliant software, there is some point you
> just have to trust someone else to not jack you.

A supplier of Free Software can never be sure that someone he doesn't even know about let alone control will decide to review his source code.

Re:what do you expect?

[SQLGuru]

I'd rather have better video support than USB support so that you play games in a virtual machine (and by games, I mean games beyond 2D games from the early 90's). If a virtual machine would support something like DirectX or OpenGL so that I could have the kids running their games in a virtual machine (and being able to install them, etc.) I would have them set up with a locked down OS with a virtual system for their games.

There are some options, but they haven't been successful for me yet. But I'm sure the technology is getting closer.

Layne

Re:Ummm... How?

[cduffy]

If you read the article, you'd know it's the license-management code. Licenses expire.

Workaround available

[fredr1k]

A workaround is possible Turn off NTP time on the host. And manually (using the VIC) change that date to one week backwards in time. Voila all set to work.

Re:Workaround available

[d_ron_218]

The only way to run a Windows domain controller in VMware is to tie its clock to the physical host's clock. And lots of things break if your domain controllers have the wrong time (Kerberos authentication, NTP across the Windows network, etc, etc). So changing the host clock would generally be a bad idea.

Re:Workaround available

[Sobrique]

I've had oodles of grief from VMs running as DCs for exactly this reason - they pick up clock skew as they're not running _quite_ in real time. And so they drift, and as soon as they hit the ... is it 5 minute? Kerberos window, your whole domain goes nuts.

Troubleshooting that one was fun.

Re:Workaround available

[exi1ed0ne]

1) edit the vmware config file to include the line:

host.cpukHz = XXXXX

Where XXXXXis your CPU in kilohertz

2) enable time sync via vmware tools

3) modify Type in HKLM\System\CurrentControlSet\Services\W32Time\Parameters from "Nt5DS" to "NoSync"

That's always taken care of clock skew issues on DCs for me.

My head hurts.

[dc29A]

My head hurts reading that article. Who the fuck wrote it? A ten year old mental retard?

It's like ............... this and VM's this VM's that (Yes, notice the spelling?). Ooooh and the cyberwarfare boogeyman! You can't even find this much Hollywood scenario fear mongering from Hollywood themselves. Oh noes! Our entire infrastructure will be killed by evil cyber terrorists because it runs on VMware!

Oh and and lovely parts like 'w/' instead of 'with'. Hey douchebag, this is not SMS, is it so hard to hit another 2 keys on your keyboard? Oh and for the love of $DEITY$, please learn basic HTML and use links so I don't have to copy paste text into the address bar.

As for Slashdot editors, why the fuck did they pick the worse possible article from the Firehose when plenty others look *WAY* more professional?

Re:My head hurts.

[dotancohen]

Even worse, he got the meme wrong. The title of the blog post should have been "All your VM are belong to us". Idiot.

Re:My head hurts.

[Anonymous Coward]

for the love of $DEITY$

That's either $DEITY or %DEITY%, please learn basic shell scripting for your platform :)

Morale: if you're gonna rant, make sure you do not make the same mistakes as the target of your rant

Re:My head hurts.

[kiwimate]

for the love of $DEITY$

That's either $DEITY or %DEITY%, please learn basic shell scripting for your platform :)

Morale: if you're gonna rant, make sure you do not make the same mistakes as the target of your rant

That's moral, which is a lesson to be learned. Morale refers to high spirits, or lack thereof, as in "his morale was crushed when he realized his error in verbiage".

Yes, it is a bug

[evilpenguin]

But the real bug is license enforcement in the first place. Why would you run the risk of making your business depend on the whims of someone else's IP policies and enforcement?

Now, I'm somewhat realistic. I know that there isn't (yet) an adequate replacement for every piece of closed proprietary software out there. But for my own business (admittedly small) I am building with nothing but GPL/BSD/Apache license code. And it is working. I don't trust closed code. Of course my software will have bugs, some of them serious. But I won't have stuff shutting down because of "license" issues. Why do people go quietly into enforced licenses? Why do people accept remote kill switches on their servers? Why doesn't this strike everyone as a crazy thing to do?

Re:Yes, it is a bug

[Waffle Iron]

And when that happen? Who cares, I'll just sue their asses, like I do whenever OTHER problems come up, and it works to recover losses

No you won't. For essentially any software product available on today's market, during installation you agree to waive your rights to recover any losses beyond the purchase price.

Re:Ummm... How?

[Darkness404]

Ah, see, another reason why free software always is better

VMWare's KB down

[Anonymous Coward]

"Temporary Maintenance - Knowledge Base

This section of the VMware website is currently unavailable while we make important user improvements and upgrades to the site. We apologize for any inconvenience this may cause."

I hope it wasn't running on a VM.

Utility computing w/o virtualization

[MarkEst1973]

VMWare licenses for ESX server cost something like $5k apiece. My company uses VMWare and I don't quite get it. We pay for expensive blade hardware ($8k each for those, not to mention the chasis), then we pay $5k per virtual server. And for what? Adding virtualization overhead to the runtime cost.

Meanwhile, in articles like this, people are showing how to run many applications and different versions within a single container. A single node in the cluster can run any application. There are always busy, keeping the hardware fully utilized. Isn't that the promise of utility computing? Rack up a bunch of cheaper (but not cheap/shoddy) servers and let your cluster go to town.

So, my question is, why are we (as an industry) embracing virtualization when apps written for a smart container (like OSGi) give the same benefits without all the additional co$t and runtime overhead?

Re:Utility computing w/o virtualization

[Cyberax]

Isolation and easy management.

Isolation of applications in OSGi containers is leaky, one bad-behaving application can bring down the whole containers.

Lightweight containers (OpenVZ, Virtuozzo) have almost no overhead and allow cool features like load-balancing of ALL applications between cluster nodes. However, all lightweight containers use the same kernel, and one kernel bug can bring down all virtual nodes.

XEN/KVM have a bit more overhead but with even more isolation (each node has its own kernel).

Re:Utility computing w/o virtualization

[Anonymous Coward]

Simple...power. Right now our datacenter is strapped for power, and power isn't cheap. Neither is cooling. For 10U and 8000 watts I can install a fully loaded blade chassis with 128 CPU cores and 1 Terabyte of RAM, attach it to a SAN and run 150 VMs in it. Or I can install 150 rack and stack servers at taking up 4 racks and 75000 watts. Let me think here...

And while I'm thinking about it, let's also remember that using VMWare gives you options like DRS and VMotion that you don't get with physical hardware. Or you can replicate your SAN to another SAN at your DR site and have a VMWare cluster waiting there for recovery. Then instead of having to do a bunch of restores to bare metal hardware, you could potentially get your servers back up and running in minutes instead of hours.

There are many, many benefits to virtualization. If there weren't then people wouldn't have been using for decades in one form or another.

"License management code..."

[John Hasler]

...Says it all, I think. Perhaps you should reconsider the ramifications of making your business critically dependent on software that contains code specifically design to make it stop working.

Consider this: to a proprietary vendor the only safe failure mode for "license management code" is one where everything stops.

Patch Tuesday

[Thelasko]

FTFA:

VC will continue to show the hosts as licensed and no errors will appear in vmkernel log file until you try to start up a new vm, reboot a vm, or reboot the host.

Um, isn't today Patch Tuesday? This could be worse than we thought.

Re:Patch Tuesday

[prandal]

Rebooting a host doesn't power down the VM.

The licence checking is done at VM power up, apparently.

License Management Server

[Comatose51]

Unless something has changed dramatically, an expired license won't bring down any already deployed VMs. It simply won't allow you to deploy undeployed ones. It doesn't shut down the VMs as the headline makes it sound nor is it a bug in the hypervisor. Yes it's embarrassing that this got out but can we have a less sensationalist headline and summary?

KVM and XEN

[kenp2002]

The Open Source Model gets a leg up again after this nonsense. A client of mine just ported all their VMs and said good bye to VMware. That's 280 VMs by the way. Thank God we had a contingency plan for switching VM providers for a DR exercise a year ago and here we go.

Management is pretty upset and I doubt we will be switching back any time soon to VMWare products after this.

On a side note this scenario did prove one thing:

Having a VM-agnostic storage makes migration easy. We changed a mount point, powered on the alternate VM host and we were off and running just that quick. We lost the ability to do live migrations for now but beyond that is was a good opporunity to see just how important an VM-agnostic disk storage array is. (I'm not the admin of those machines but I believe we are using iSCSI).

On my side though I had about 50 scripts tapping VMWare via PERL but I guess I can start building workarounds now... No more batch submission and dynamic routing for a week or two... The part I hate the most was I had a nice script to take a batch submission and if necessary migrate a utility node to bigger hardware to accomidate the batch... pisses me off but what can I do, thank you Vmware, that aquisition seems to be improving your product as much as when Symantec aquired Ghost Corp!

No! Don't set the time back!

[Animats]

VMware is suggesting setting the system time backwards to work around their license manager problem. That's a desperation move. Not only will it mess up everything from Kerberos to CVS to "make", if you're running certain licensed software, in particular software licensed via FlexLM, that software will stop working. FlexLM will disable your licenses if the clock goes backwards by more than 24 hours. Now your expensive high-end software protected by FlexLM (Rational, Avid, Matlab, National Instruments, ANSYS, Cisco Unity, Clearcase, Nokia network management, etc.) will stop working. Setting the clock forward again may not re-enable it, either; there's tamper detection.

Also, if you have server/client licensing with FlexLM, or multiple license servers, and the clocks disagree significantly, FlexLM gets suspicious and turns licenses off.

Re:Nice..

[hany]

You know that you should read ./ before you do any actual work. Don't you?

:)

NOT the *OpenSource Edition*

[DrYak]

Support for USB, iSCSI and RDP (along with USB-over-RDP) are only available in the closed source variants of VirtualBox.
The opensource edition of Virtual Box doesn't have them.

Also the USB support may lock the system when in fast emulation/patching/ring-2 mode, and only works flawlessly when using the slower mode with virtualisation CPU extensions (my brother tried using it to get old USB hardware accessible when moving to Vista 64 but since then he ended up buying newer hardware)

I Am So Dead

[Hasai]

*sigh*

Well, it's for real. I've confirmed it here, and my whole data center is affected.

It's time like this when I wish I hadn't left the Army; at least there, you can shoot back.

This is going to be one hell of a long night. :(

Bad company

[PingPongBoy]

but if I don't know what the hole looks like, I can't carve a peg to fit it

There are some I know who will put their pegs into any hole