Massive VMware Bug Shuts Systems Down

mattmarlowe writes "Imagine if Red Hat released a version of Linux, and after it was deployed, customers noticed that any processes with a start date of today would refuse to run? Well, that's what happened to VMware — a company that wants nearly all server applications running in virtual machines within a matter of years." Supposedly a fix will be available ... in 36 hours.

Comments

License Management Software!?

[bigtallmofo]

I don't get license management measures in software that is only going to be used by major corporations.

If someone wants to run virtual machines at home or in a small business, they're likely going to be more than satisfied with VMWare Virtual Server (formerly GSX) and wouldn't even consider the much more complex ESX.

In a major corporation, fear of massive fines and prosecution is enough to stop them from pirating your software. Hardware dongles, software license managers and the like only hurt your paying customers.

Re:License Management Software!?

[db32]

Exactly. It is a tremendous pain in the ass to track all the stupid license keys and crap in use. Departments frequently need software specific to only their department and outside the scope of normal IT support stuff. Phone numbers, licenses, etc. God forbid any of those companies get purchased or go under, then you are stuck with expensive software that you cannot recover.

The call home variety is extremely infuriating. On top of whatever nonsense key/activation crap you have to go through, you have to put up with it trying to call home or deactivating itself. MS isn't the only guilty party in this, but those bastards certainly made the situation much worse.

Re:License Management Software!?

[morgan_greywolf]

Exactly. Most large companies usually have an entire person, and sometimes multiple people dedicated to nothing but license management.

What a colossal waste of money.

Re:License Management Software!?

[rudeboy1]

Good god do I hear you, brother. I work IT for a legal firm. So many little apps no one else in IT has ever even heard of. And most of them, you're talking to the same guy for support that developed it, and filled the sales order. Out of his basement or garage. Multi-million dollar a year law firm, and it can be brought to its knees if one of our obscure applications goes down and needs support, and the one guy that can support it is out taking his kids to soccer practice.

I'm looking at you North Winds Software. I'll BUY a support contract! If you offered such a thing. If you answered the phone.

I need to go back to bed. :(

Re:License Management Software!?

[_merlin]

Having administered ESX, I can say the license management is useful for one thing: it helps you ensure you aren't exceeding what you're licensed for. For example, if you aren't licensed for multi-processor boxes, it will complain until you get a valid license. If nothing else, it gives you some confidence that you will pass an audit.

License management is also useful for things like MATLAB and OPNET that are licensed per concurrent user: you can install on as many machines as you like, but they need to be able to talk to your license server (not that this is _your_ license server on your network - it isn't "calling home") to ensure that the number of concurrent users is below the maximum allowed. That way, if say, everyone needs to be able to run OPNET occasionally, but not very often, everyone can install it, but you only need to pay for a few licenses. You know you aren't exceeding your licenses because it won't let you launch more instances than you're allowed simultaneously. If your users regularly complain that they can't fire up OPNET due to lack of licenses, you pay for a few more seats.

On the other hand, I can't stand software that calls home to ensure that it's "genuine" a la Windows Vista, or those stupid CD copy protection schemes. That's bullshit. Things like that make more work for a sysadmin, not less. I only like license management when it helps me, the admin; I don't care what it does or doesn't do for the software vendor. I'm a selfish pig, I know.

Another thing I can't stand is things like Rational Purify where they attempt to count your "activations" at their end: when you install Purify, it increases the installed count in IBM's system, and decreases it when you uninstall. If the IBM server thinks you're using all your licenses, you can't install. Too bad people always forget to uninstall Purify before wiping their computers for a clean OS install (or scrapping the computers)! And don't get me started on how bad it is to deal with IBM's phone support. This is one copy protection scheme that I do bypass: I install Purify in a VMware virtual machine, snapshot it, uninstall Purify, and roll the virtual machine back to the snapshot. That way, Purify will work in the virtual machine, but IBM's servers will think I haven't used any of my licenses. Also, I can make copies of the virtual machine for multiple people to use. It's easier for me to track the licences than put up with a crap license management scheme.

Re:License Management Software!?

[supersnail]

Actually its quite a common policy in MegaCorps to reject software that require machine specific or expiring license keys for use in "Mission Critical" applications.

The backup server not having the correct licenses is one of the biggest risks in a Disaster Recovery.

Migration to newer better hardware also becomes a nightmare where license keys are involved -- what do you mean the new server doesnt have centronics port for the dongle?

Its also screws up the companys virtualisation strategy as you have no idea whether a given license scheme will work in inside a VM or not.

Do like the Fortune 500 and just say no to runtime licenses.
       

Re:License Management Software!?

[Gordonjcp]

I'm looking at you North Winds Software. I'll BUY a support contract! If you offered such a thing. If you answered the phone.

There's an Ask Slashdot for you. Is there something out there that can replace this magic bit of software? Is anyone interested in writing an Open-Source equivalent?

Re:License Management Software!?

[swabeui]

I'm looking at you North Winds Software. I'll BUY a support contract! If you offered such a thing. If you answered the phone.

North Winds Software? Just a WILD guess... is this 'software' based on MS Access? I wonder where they got the company name from...

Re:License Management Software!?

[fr175]

I really don't think the fines would keep large corporations in line. look at all the stuff you see big business doing that they know is illegal and that they know will land them big fines if they get caught. Software piracy is no different. In fact it's probably easier to use a pirated piece of software than it is to dump illegal chemicals or defraud investors. You can manage the exposer.

Having acted in an advising capacity on a software license management project currently underway at one of the worlds largest financial institutions (400k employees), I disagree. Purchased software is an asset on the books and needs to be tracked. Pirated software is a risk and even the largest companies will occasionally be brought to court for "over implementation."

The main hurdle with Software Asset Management (SAM) is the complexity of the licenses involved, and the multitude of way in which it can be obtained. Some examples: is the license perpetual or subscription based; is it a "named user" license or is it assigned to the org; does it include maintenance (upgrade rights); if it includes maintenance is the maint co-termed with the other licenses that the org owns; if it includes maintenance, what was the most current version at the time the maintenance expired; does the current version allow for "downgrades" and how many version prior can be downgraded; what previous versions qualify for an upgrade license and which would need a full new version; can the licenses be transferred within the org; can they be transferred globally; does the license allow for home use; does the license allow for portable device use; just to name a few.

If large corporations were willing pirates, you would not see them making their annual multi-million dollar payments to Microsoft for their Enterprise Agreements. You wouldn't see them spending millions on risk management/mitigation consultants or conducting their own software audits. There are people out there getting paid piles of cash to implement a working SAM system.

It's unavoidable that a large corporation will be under-licensed. However, they spend big bucks to mitigate the risk that this opens them up to.

Re:License Management Software!?

[alohatiger]

What is a "disadvantaged business" anyway, and why would someone actually use that as a sales point?

Government work. Some government contracts require a percentage of the work to be done by minority/women/veteran/disadvantaged owned businesses.

Re:License Management Software!?

[tsstahl]

I'm looking at you North Winds Software. I'll BUY a support contract! If you offered such a thing. If you answered the phone.

Um, isn't North Winds the name of the company that comes with the sample Access database? They're not real, you know... ;)

Re:License Management Software!?

[MBGMorden]

and as far as everyone else in the business is concerned, any failings in the product is the IT department's problem not theirs

This is true, and particularly frustrating. We recently have converted from an (old, but very functional and stable) 20+ year old COBOL program to a new Windows application in our organization. This is a Visual Basic application that if I'm being kind I'd say is a kludge held together by the electronic equivalent of duct tape and glue. The thing is junk and crashes ALL THE TIME. IT didn't pick this app though - we just get stuck supporting it. However, no amount of explanation can convince these people that the program crashing is not IT's fault. We can reinstall it as many times as they ask for it. We can update everything on their computer. We can buy them a new computer. But the basic fact is the program you bought is crap and full of bugs and nothing IT does is going to make it stop crashing and screwing up data.

Sadly, this is a hard fact to make users accept.

Re:License Management Software!?

[Anonymous Coward]

I'm on the other side of the divide---a tiny company that's not too much more than a guy in his garage (just a few of us), and frankly, I agree with you. I'm astonished at the way we do things, even though we sell to huge firms (including big law firms, like yours). Part of it is just size---we don't have the people or skills to do all the safety, security and support steps a big corporation would. Still, freaks me out that the crap I wrote is out there being used to do important things by important people who don't realize how dumb the guy who wrote their software actually is.

Re:License Management Software!?

[IntlHarvester]

The problem is that every industry has a few of these super-specialized vertical apps that come from one-guy software companies. Most of them are fairly simplistic Access/FoxPro type things, the hard part was implementing all of the business rules.

I've worked with a few companies that recreated their software package in-house (because they needed specific customization the author wouldn't provide), and it's never as cheap or easy as it might seem superficially.

Re:License Management Software!?

[SpaceLifeForm]

Well, that explains the lack of support.

Can't start processes?

[oldspewey]

any processes with a start date of today would refuse to run? Supposedly a fix will be available... in 36 hours.

Good thing the fix will be available tomorrow, because if it was available today nobody would be able to run the update process

what do you expect?

[larry bagina]

Who knows what else is lurking in their code base? Certainly not me or you -- we can't see it. We're at their mercy to find and fix problems.

I stick to virtualbox. I'm not going to pretend I've audited the source code, but if I need to, I can.

Say YES to freedom.

Re:what do you expect?

[Anonymous Coward]

Then give me USB support in VirtualBox. Cause I kinda need that the most.

Re:what do you expect?

[ray-auch]

USB license dongle for the application software running on the VM.

Seriously. Last week.

Re:Ummm... How?

[cduffy]

If you read the article, you'd know it's the license-management code. Licenses expire.

Workaround available

[fredr1k]

A workaround is possible Turn off NTP time on the host. And manually (using the VIC) change that date to one week backwards in time. Voila all set to work.

Re:Workaround available

[d_ron_218]

The only way to run a Windows domain controller in VMware is to tie its clock to the physical host's clock. And lots of things break if your domain controllers have the wrong time (Kerberos authentication, NTP across the Windows network, etc, etc). So changing the host clock would generally be a bad idea.

Re:Workaround available

[Sobrique]

I've had oodles of grief from VMs running as DCs for exactly this reason - they pick up clock skew as they're not running _quite_ in real time. And so they drift, and as soon as they hit the ... is it 5 minute? Kerberos window, your whole domain goes nuts.

Troubleshooting that one was fun.

Re:Workaround available

[exi1ed0ne]

1) edit the vmware config file to include the line:

host.cpukHz = XXXXX

Where XXXXXis your CPU in kilohertz

2) enable time sync via vmware tools

3) modify Type in HKLM\System\CurrentControlSet\Services\W32Time\Parameters from "Nt5DS" to "NoSync"

That's always taken care of clock skew issues on DCs for me.

My head hurts.

[dc29A]

My head hurts reading that article. Who the fuck wrote it? A ten year old mental retard?

It's like ............... this and VM's this VM's that (Yes, notice the spelling?). Ooooh and the cyberwarfare boogeyman! You can't even find this much Hollywood scenario fear mongering from Hollywood themselves. Oh noes! Our entire infrastructure will be killed by evil cyber terrorists because it runs on VMware!

Oh and and lovely parts like 'w/' instead of 'with'. Hey douchebag, this is not SMS, is it so hard to hit another 2 keys on your keyboard? Oh and for the love of $DEITY$, please learn basic HTML and use links so I don't have to copy paste text into the address bar.

As for Slashdot editors, why the fuck did they pick the worse possible article from the Firehose when plenty others look *WAY* more professional?

Re:My head hurts.

[dotancohen]

Even worse, he got the meme wrong. The title of the blog post should have been "All your VM are belong to us". Idiot.

Re:My head hurts.

[Anonymous Coward]

for the love of $DEITY$

That's either $DEITY or %DEITY%, please learn basic shell scripting for your platform :)

Morale: if you're gonna rant, make sure you do not make the same mistakes as the target of your rant

Re:My head hurts.

[kiwimate]

for the love of $DEITY$

That's either $DEITY or %DEITY%, please learn basic shell scripting for your platform :)

Morale: if you're gonna rant, make sure you do not make the same mistakes as the target of your rant

That's moral, which is a lesson to be learned. Morale refers to high spirits, or lack thereof, as in "his morale was crushed when he realized his error in verbiage".

Yes, it is a bug

[evilpenguin]

But the real bug is license enforcement in the first place. Why would you run the risk of making your business depend on the whims of someone else's IP policies and enforcement?

Now, I'm somewhat realistic. I know that there isn't (yet) an adequate replacement for every piece of closed proprietary software out there. But for my own business (admittedly small) I am building with nothing but GPL/BSD/Apache license code. And it is working. I don't trust closed code. Of course my software will have bugs, some of them serious. But I won't have stuff shutting down because of "license" issues. Why do people go quietly into enforced licenses? Why do people accept remote kill switches on their servers? Why doesn't this strike everyone as a crazy thing to do?

"License management code..."

[John Hasler]

...Says it all, I think. Perhaps you should reconsider the ramifications of making your business critically dependent on software that contains code specifically design to make it stop working.

Consider this: to a proprietary vendor the only safe failure mode for "license management code" is one where everything stops.

Patch Tuesday

[Thelasko]

FTFA:

VC will continue to show the hosts as licensed and no errors will appear in vmkernel log file until you try to start up a new vm, reboot a vm, or reboot the host.

Um, isn't today Patch Tuesday? This could be worse than we thought.

Re:Patch Tuesday

[prandal]

Rebooting a host doesn't power down the VM.

The licence checking is done at VM power up, apparently.

KVM and XEN

[kenp2002]

The Open Source Model gets a leg up again after this nonsense. A client of mine just ported all their VMs and said good bye to VMware. That's 280 VMs by the way. Thank God we had a contingency plan for switching VM providers for a DR exercise a year ago and here we go.

Management is pretty upset and I doubt we will be switching back any time soon to VMWare products after this.

On a side note this scenario did prove one thing:

Having a VM-agnostic storage makes migration easy. We changed a mount point, powered on the alternate VM host and we were off and running just that quick. We lost the ability to do live migrations for now but beyond that is was a good opporunity to see just how important an VM-agnostic disk storage array is. (I'm not the admin of those machines but I believe we are using iSCSI).

On my side though I had about 50 scripts tapping VMWare via PERL but I guess I can start building workarounds now... No more batch submission and dynamic routing for a week or two... The part I hate the most was I had a nice script to take a batch submission and if necessary migrate a utility node to bigger hardware to accomidate the batch... pisses me off but what can I do, thank you Vmware, that aquisition seems to be improving your product as much as when Symantec aquired Ghost Corp!

Re:Utility computing w/o virtualization

[Anonymous Coward]

Simple...power. Right now our datacenter is strapped for power, and power isn't cheap. Neither is cooling. For 10U and 8000 watts I can install a fully loaded blade chassis with 128 CPU cores and 1 Terabyte of RAM, attach it to a SAN and run 150 VMs in it. Or I can install 150 rack and stack servers at taking up 4 racks and 75000 watts. Let me think here...

And while I'm thinking about it, let's also remember that using VMWare gives you options like DRS and VMotion that you don't get with physical hardware. Or you can replicate your SAN to another SAN at your DR site and have a VMWare cluster waiting there for recovery. Then instead of having to do a bunch of restores to bare metal hardware, you could potentially get your servers back up and running in minutes instead of hours.

There are many, many benefits to virtualization. If there weren't then people wouldn't have been using for decades in one form or another.

No! Don't set the time back!

[Animats]

VMware is suggesting setting the system time backwards to work around their license manager problem. That's a desperation move. Not only will it mess up everything from Kerberos to CVS to "make", if you're running certain licensed software, in particular software licensed via FlexLM, that software will stop working. FlexLM will disable your licenses if the clock goes backwards by more than 24 hours. Now your expensive high-end software protected by FlexLM (Rational, Avid, Matlab, National Instruments, ANSYS, Cisco Unity, Clearcase, Nokia network management, etc.) will stop working. Setting the clock forward again may not re-enable it, either; there's tamper detection.

Also, if you have server/client licensing with FlexLM, or multiple license servers, and the clocks disagree significantly, FlexLM gets suspicious and turns licenses off.