Security Flaw Hits VAserv; Head of LxLabs Found Hanged

Keldrin_1 writes "The discovery of 24 security vulnerabilities may have contributed to the death of the chief of LxLabs. A flaw in the company's HyperVM software allowed data on 100,000 sites, all hosted by VAserv, to be destroyed. The HyperVM solution is popular with cheap web hosting services and the attacks are easy to reproduce, which could lead to further incidents."

Comments

Narrow escape

[sakdoctor]

Just closed an account with VAserv last week for no particular reason.
I hardly ever do things for "no particular reason" so it must have been my spider sense.

Will this be a case of good bye reputation, or no publicity is bad publicity?

Re:Narrow escape

[TheRaven64]

An SQL injection vulnerability, via the web, gained hypervisor-level access to their system. Let me say that again; a web server, an SQL database, and a web app were all running with sufficiently-high privilege that a vulnerability in one caused arbitrary-code execution at the hypervisor level. Anyone who doesn't immediately start worrying when they see that kind of lack of privilege separation has absolutely no business running a VPS business. I definitely won't be putting any business VAserv's way in the future...

Mixed feelings

[JeffSpudrinski]

You can't truly blame Milw0rm for a person being depressed and committing suicide.

However, reading their security notes on it, they did hear back from the developer...they simply declared that it didn't happen fast enough and decided unilaterally that the "Vendor appears uninterested".

I have very mixed feelings on security firms releasing exploits to the public just to try and get results. In my (admittedly limited) experience, more bad has come from releasing exploits publicly than good.

-JJS

Re:Mixed feelings

[Zashi]

Actually thanks to cyber-bullying laws I think you can.

Re:Mixed feelings

[CarpetShark]

You can't truly blame Milw0rm for a person being depressed and committing suicide.

No, you truly can. You can't blame it for 100% of the problem, but without doubt, people who make viruses are preying on others. What outcome to you expect, when those preyed upon are already struggling just to get through the day and raise their kids or whatever?

Re:Mixed feelings

[Todd Knarr]

The problem with that approach is that the day after you announce that the exploit exists the company's going to sue you for defamation and libel. They're going to use the fact that you didn't provide any evidence to support your claim against you as evidence that you don't have any evidence and are lying specifically to harm their reputation. And one of the first things they're going to ask for is an order barring you from libeling them any further, which is going to prevent you from disclosing anything to help clear your reputation. By the time you get the lawsuit untangled you won't be able to demonstrate that the vulnerability really existed (the fix will have been quietly added during a regular update and your exploit won't work anymore) and you'll end up with the trashed reputation.

My position:

1. Contact the vendor with full details. Give them working exploit code so they can verify for themselves that it really works. They get 2 weeks to respond confirming they've received your material, and no more than an additional 4 weeks to at least notify their customers of the problem.
2. If the vendor fails to meet either of those deadlines, publicly release a full report on the problem including sufficient details and/or code for other parties to verify the correctness of your claims and any known workarounds or steps that can be taken to mitigate the vulnerability. By this point the bad guys already have all this information, now the affected users also have it and can take steps to protect themselves.
3. Don't threaten. Don't bluff. Your position should be Londo's: "Actually, now that you mention it...".

Re:Mixed feelings

[nxtw]

Two weeks is not nearly enough time to even decide if something like this is worth looking at, let alone find a fix, develop it, test it, implement it, and push it to all clients

Are you serious?

According to milw0rm, whoever responded didn't even access the details of the vulnerabilities - after two weeks. Nor did they provide any contact information. It would only take a few minutes to skim through the details, and it should have been immediately apparent that the vulnerabilities described could be serious. But they didn't read the details at all.

Assuming milw0rm did contact the correct person/people at LXLabs, they clearly has no interest in the security of their product(s).

Re:Mixed feelings

[nxtw]

Someone sends a random, out-of-the-blue email saying "hey we hax0red your code, lol" and you expect the recipient to pop tall and check out their site immediately? Are you serious?

No one looked at the details of the vulnerabilities for two weeks, after they claimed they would look at it and after they claimed they would respond in a few hours.

And what contact information was needed? Obviously Milw0rm talked with someone at the company, so they already had contact information.

In a business context, it is customary for people to sign their emails with (at least) their name.

However, I am willing to state, without reservation, that Milw0rm are a bunch of asshats who deserve to be sued into oblivion over their callous disregard for the safety of the customers using this software.

If anyone callously disregarded the safety of LXLabs' customers, it was LXLabs. Milw0rm's disclosure aside, it's LXLabs who made a product with such severe security issues and LXLabs who made ridiculous claims about the security of their product.

Most of the people hurt by this had no control over the software getting fixed, had no idea there was a problem until it was too late to do anything about it, and were completely innocent of any mistakes.

LXLabs' customers chose the product to begin with! If the product is indeed this insecure, the customers are certainly not innocent, as they have failed to thoroughly evaluate the product.

And yet Milw0rm doesn't care one fig about those people and just releases code that sends their lives and businesses into a tailspin.

It is not milw0rm's responsibility to care for LXLabs' customers. That's LXLabs' job.

It may have been genetic

[BadAnalogyGuy]

His sister and mother both committed suicide by hanging 5 years ago. He may have had a genetic propensity towards suicide.

Culturally, Indians have a very heavy emphasis on honor and responsibility. The failure of the software is only the outermost layer of true damage. Each of those compromised VMs is a failure to satisfy a customer at best, and a grave violation of the trust between vendor and customer.

When it comes to suicide, why hanging? It seems like a really hard way to go. Maybe the person wants to suffer to pay back his debts before death.

Re:It may have been genetic

[fuzzyfuzzyfungus]

Not really. Plenty of people commit suicide in their late 20s or later, which leaves a decent slice of breeding time, depending on how early you start. Particularly with modern social structures(where orphans are incrementally less likely to starve in Dickensian workhouses) you can fairly easily pump out surviving children at greater than replacement rate, even if you are dead by 30.

Also, a "propensity" toward suicide isn't necessarily fatal, depending on life conditions. If you don't run into much serious stress, a tendency to respond badly to stress is largely harmless. If your son gets it and runs into a series of nasty business reversals, it'll bite him.

Re:It may have been genetic

[Anonymous Coward]

Sadly, I've never seen that level of dedication to quality in anything touched by an Indian outsourcing provider. It's always a pile of crap that you spent twice as long overspeccing to make sure they didn't mess up, then whatever came back was so broken that you spent twice as long as it would have taken to do it right the first time trying to fix it. You can't just wipe it and start over because whoever the bright bulb was who insisted on outscourcing to begin with will have a lot of political clout invested in not looking like the weenie they actually are.

Posting anonymously because, well, I'm a coward.. but I speak the truth (as I have seen it).

Re:Who else?

[FishWithAHammer]

Many/most (cheapvps, fsckvps, etc.) are reselling VAserv stuff, so a lot have been hit hard.

If they're using HyperVM, stay the hell away.

Re:VM Attacks

[Zocalo]

Actually, this has almost nothing to do with attacking VMs and more to do with the simple fact that LxLab's code is an extremely poorly written piece of crap from a security standpoint that leaves the VM wide open to attack. Having read through the 24 sample exploits when they were first published on milw0rm, the errors are pretty damn fundamental and indicate a complete ignorance of many of the established best practices in secure coding. It was just a matter of time before one of LxLab's users got hit and hit hard; frankly I'm surprised it took so long.

The only thing that I found surprising about the attack on VAserv is that the perpetrator decided to blow away the servers instead of subvert them for sending spam or hosting related websites; 100,000 web hosts have got to be worth quite a few dollars on the right market. While it sucks to be VAserv or one of their customers right now, it's probably better things went this way than the alternative for everyone else. Of course, it's just a matter of time before the next users of LxLabs HyperVM gets hit - if they haven't been already - and at least some of them are almost certainly going to be end up doing something less than legitimate.

Re:Well

[Comatose51]

Agreed but I think that kind of situation or attitude is more prevalent than we think. People build their lives around different things. Their "work" (as in the product of their effort, not as in what they do from 9 to 5) becomes their lives. This is especially true of the creative types such as artists and writers but also software engineers. In many ways, software engineering or engineering in general is a hybrid between the arts and the sciences with room for creativity and personal touches. I work with a good group of engineers who are very passionate about their work, much more so than our paychecks can account for. I've seen the same passion turn into despair in bad times as well. Engineers also compound this problem by not being the most social people in the world. Having a network of people to connect to can really soften the pain when things don't go well. Most engineers don't commit suicide but the rate of burning out is rather high.

Re:Well

[espamo]

TFA: "Ligesh [from LxLabs] was also still coming to terms with the suicides by hanging of his sister and mother five years ago."

I suspect that this was the result of a lot of bad things going on in his life, and not just because of the software issues.

And very likely a genetic predisposition to suicide as well.

Summary of Vunerabilities

[BrittanyGites]

Summary from http://www.milw0rm.com/exploits/8880 seems pretty serious but quite difficult to fix all of them in 2 weeks.

      Timeline :

      05/21/2009 - sent initial email to vendor with a link to a private
                                resource for viewing various kloxo hiab575
                                vulnerability info
      05/23/2009 - received the following: "Thanks for the info. I will
                                review this and let you know." (no signature)
      05/30/2009 - sent an email asking if there were any updates
      06/01/2009 - received the following: "Sorry for the delay. I am
                                currently looking into this, and will reply in a couple
                                of hours time." (no signature)
      06/04/2009 - nothing heard from vendor, and the private resource
                                containing the vulnerability info still does not
                                appear to have been accessed

      2 weeks have passed since the initial notification. Vendor appears
      uninterested.

      ISSUE 1 - uid/gid reuse
      ISSUE 2 - unprivileged port use
      ISSUE 3 - default passwords
      ISSUE 4 - useradd string in the process list
      ISSUE 5 - XSS
      ISSUE 6 - remotely create partially user controlled file names
                            and directories. Locally append uncontrolled data to
                            any file
      ISSUE 7 - local users can take control of any file or directory
      ISSUE 8 - local users can take control of any file or directory
      ISSUE 9 - local users can overwrite any file on the box
      ISSUE 10 - yet another symlink attack for local users
      ISSUE 11 - metachar injection, local command execution as root
      ISSUE 12 - web stats world readable password hashes
      ISSUE 13 - local users can overwrite any file on the box
      ISSUE 14 - metachar injection, local command execution as root
      ISSUE 15 - remotely block any - or every - IP addr in hosts.deny
      ISSUE 16 - remote CPU and mem usage DoS
      ISSUE 17 - local users can truncate and control any file
      ISSUE 18 - just 2 more symlinks to own any file on the box
      ISSUE 19 - file manager, view and edit any file
      ISSUE 20 - file manager PT II
      ISSUE 21 - file manager PT III
      ISSUE 22 - local user symlink attack
      ISSUE 23 - local user symlink attack (last one)
      ISSUE 24 - sql injection in the "Forgot Password" form

Re:Hackers = murderers?

[Anonymous Coward]

I was wondering what milw0rm would get from publishing it openly? It could give out information on a as-needed basis. Example: If LxLabs didn't fix it on time but a user wanted to, milw0rm could announce that they've found some exploits and they could give it out with a three way verification.

But publishing it openly and giving it to script kiddies to play with is totally irresponsible. For that matter, vulnerability notification blackmailing is something that nobody is prosecuting under the law today.

Re:There's yer problem...

[Glendale2x]

I disagree; it should logically follow that a company should have some kind of disaster recovery plan other than "Oops, it's all gone, but how about a few months of free service?" If that's what customers want and I could get away with then damn, I've been wasting time and money keeping disaster recovery backups offsite. I'm not talking about backups like customers accidentally deleting files, but loss of service due to events beyond your control.

Yes, you should have copies of your own stuff, the more the better. For vahost even if the "oh crap" backup was a week old that would have been better than the total loss they're selling as "not our fault we dun got hacked".

Re:Well

[svnt]

I'd be like Kevin Spacey in American Beauty...

You did see the entire movie, right?

Notable characteristics of Kevin Spacey's character: in the middle of a mid-life crisis, hated by his daughter, hates his wife, has sexual contact with a minor. Oh, and he happens to work at a fast food restaurant.

This is just a friendly suggestion, but before you tell this story to people you actually know, maybe refine your role model selection a little?