Researcher Finds Critical OpenVPN Bug Using Fuzzing (zdnet.com)

"Guido Vranken recently published 4 security vulnerabilities in OpenVPN on his personal blog," writes long-time Slashdot reader randomErr -- one of which was a critical remote execution bug. Though patches have been now released, there's a lesson to be learned about the importance of fuzzing -- bug testing with large amounts of random data -- Guido Vranken writes: Most of these issues were found through fuzzing. I hate admitting it, but...the arcane art of reviewing code manually, acquired through grueling practice, are dwarfed by the fuzzer in one fell swoop; the mortal's mind can only retain and comprehend so much information at a time, and for programs that perform long cycles of complex, deeply nested operations it is simply not feasible to expect a human to perform an encompassing and reliable verification.
ZDNet adds that "OpenVPN's audits, carried out over the past two years, missed these major flaws. While a handful of other bugs are found, perhaps OpenVPN should consider adding fuzzing to their internal security analysis in the future."

Guido adds on his blog, "This was a labor of love. Nobody paid me to do this. If you appreciate this effort, please donate BTC..."

Comments

Donate

[war4peace]

We should donate TitCoin, CannabisCoin, PotCoin and the like. BTC is too mainstream.

Re: Donate

[K. S. Kyosuke]

Why do you prefer Titcoin over Buttcoin?

The power of brute force

[Dutch Gun]

Fuzzing is essentially harnessing the power of our modern computational power in a brute force fashion, combined with the knowledge that many errors (especially crashes), by nature, can be leveraged into an exploit.

In my own scripting language project, I have two fuzz tests I perform - I first fuzz a set of source scripts, and in another test, I fuzz a set of compiled bytecode, which exercises both the lexer/parser and runtime interpreter phases. I didn't even bother with a library either, just a small routine that randomly swaps and corrupts source from the original. It's really amazing how simple something like that will catch so many bugs.

Honestly, I was implementing this just for the sake of robustness. No one but me is using the library yet, and it's just for local use in my game engine. But if you're connected to the internet in any way, there's really no excuse these days for not having a set of fuzzing tests you regularly run during your normal regression testing, and there are some great libraries available to help do this. You can even leverage Google's massive computational resources for testing for free (perhaps even get paid a small bounty) if your project is important enough, which OpenVPN certainly is. Hopefully the OpenVPM devs/maintainers take note of this and make this happen, and we'll all be more secure for it.

BTW, if you ever want to read about an incredibly comprehensive test and regression suite, check out SQLite's description of their testing methodology: https://www.sqlite.org/testing...

Re:The power of brute force

[phantomfive]

What do you use to do your fuzzing? Did you write a custom set of tools?

Re:The power of brute force

[Dutch Gun]

Nothing so grandiose. Just small class that randomly corrupts bytes in a memory buffer or string. Basically, it first chooses a random 0-100% odds of corruption, then iterates through each byte in the buffer, and if another random value is greater than the corruption odds, I replace the existing byte with a random one. Stupidly trivial, but still caught a number of crashes in rare corner cases I probably never would have found any other way.

It took a tiny bit more work to set up the test scenario, which was to create an array of sample scripts (called the "corpus", as I understand it), then loop through those in a parallel array (in two phases, as I mentioned), starting with a known seed value so I can reproduce a crash as needed. Not counting the embedded corpus scripts, it was less than 150 lines of code.

Re:The power of brute force

[phantomfive]

Nice.

Re:The power of brute force

[AHuxley]

The tools exist or find a very, very smart person.

Re:The power of brute force

[phantomfive]

Read the real answer

perhaps, perhaps, perhaps.

[darthsilun]

...perhaps OpenVPN should consider adding fuzzing to their internal security analysis in the future."

This is FOSS software. If someone is doing fuzzing testing, they should jump in and contribute it. I can't imagine that the OpenVPN project, or any other FOSS software project would say no to such a contribution.

Those who stand around and offer lame suggestions like "$project should do this" are useless IMO.

I'm currently paid to work on FOSS software. There's a lot of things we need to do. There aren't enough resources to do all of them. One thing we definitely don't need are "helpful" comments like this from the peanut gallery.

Re:perhaps, perhaps, perhaps.

[Dutch Gun]

Yes, how dare a security researcher make a suggesting that might help a project find and mitigate future bugs after discovering some very serious issues using the aforementioned techniques. I mean, it's not like security is a core feature of a VPN anyhow. I'm sure they have much better things to do with their time than to harden their library using well-known methodologies that many others are using to enhance security and robustness their products.

Re: perhaps, perhaps, perhaps.

[ShanghaiBill]

Fuzzing only exposes things. Fixing them could require starting from scratch.

Most of the exposed bugs are in input validation. Once you have a repeatable test case, those sorts of bugs are usually pretty easy to fix.

Re: perhaps, perhaps, perhaps.

[Entrope]

You can contribute the scripts and configuration to run fuzzers. Presumably the developers know how to fix crashes, but they might not have the same level of expertise in setting up a fuzzer as the prospective contributor.

Re:perhaps, perhaps, perhaps.

[LostMyBeaver]

Security researchers are generally quite useless. When the report came out that OpenVPN was blessed as being secure by some high profile security researchers, I immediately went to the code base and within about 20 minutes just left as I found from a very brief review of the code an endless pile of likely attack vectors for the code. The only reason by the code was considered secure is because a security researcher ran nmap or Kali against it and it held up.

I turned up far more holes in that code through visual analysis for 20 minutes than this guy did fuzzing with far more resources. The OpenVPN code is among some of the worst code I've come across in a while. It's almost bad enough to be a VMware linux guest kernel driver.

I would have started making bug reports but honestly, that code is roughly what you'd expect from a research project for testing new ciphers. The amount of time it would take to list just the problems in their certificate code alone and provide meaningful test data would be far beyond what I'd be willing to invest in a piece of code that takes security so lightly.

I would say that OpenVPN is still many steps away from being ready for fuzzing. It would profit far more from a decent code review by people who understand programming theory, then by people who understand code security. It would help a lot if people who understood certificates AND programming looked as well.

So... while darthsilun is a bit of an ass with his wording, I'll say this, the poster begging for bitcoins for writing a 10 line script to try and crash OpenVPN really isn't being too helpful by saying "they should do this" where in reality, he should download the source code, add his test to the code, then check it in. This is how open source works.

If he believes the project should do this, then he should join the project. But simply writing a fuzzing script and tossing it at the code isn't productive. It's like standing and saying
    "Someone should help this little old lady cross the street"
    "Why don't you help her"
    "I believe it would be better if I simply stand here and watch for other little old ladies so I can say someone should help them"

OpenVPN is crap code. it's an endless pile of :
  - Linux kernel style shit C code with no input validation,
  - Absolute dependence on meaningless and generally ignored return codes
  - Spaghetti crap of "look mom, I wrote my own string compare function"
  - Function before anything else
  - Wow, we do certificates!!! Do you actually validate the certificate while parsing? Nope... we'll just assume that no one but nice people will send us certificates

Re: perhaps, perhaps, perhaps.

[Zero__Kelvin]

Give us some examples so we can review your claims.

Re: perhaps, perhaps, perhaps.

[darthsilun]

One thing we definitely don't need are "helpful" comments like this from the peanut gallery.

Ahh, the good old snobby, elitist, holier than thou, "fuck your suggestions, it's open source,

Coming from a snobby elitist holier than thou "do what I tell you and shut the fuck up". The irony is so thick you can cut it with a knife.

We already know we need to do stuff like run fuzzers. Suggestions like that are not the least bit helpful you stupid dumbfuck.

When people have constructive contributions they want to give us, we help them integrate them. When nitwits like you shout at us from the peanut gallery we ignore you.

In case you were wonder why we're ignoring you

OK whew, authenticated users escalating/exploiting

[Wrath0fb0b]

the three server flaws require the attacker be authenticated in order to exploit.

That's the good news and the bad news. It's good because it means script kiddies aren't going to be drive-by-exploiting every old unpatched DD-WRT, and that generally many of us will be kinda safe.

It's bad news because it's a huge deal both for VPN providers (kind of the obvious case) but also in the context of giving an attacker with RCE on a single client a huge vector to move laterally throughout a corporate network. I'm sure the original audit focused the majority of attention on the authentication code, but that leaves the rest of the threat model under audited.

Re:Cheers

[turbidostato]

"There are still 0-day exploits in the wild and they will be guarded well by those who discover them. Intelligence agencies will vaccinate their own systems to the exploit"

They probably shouldn't, except for their most valuable targets (if any: they most probably should let the systems go unpatched but look for a remediation elsewhere).

You don't think our enemies are hand on hand, right? They probably (but not surely) will have developed their own hacks and the one way to be sure if the enemies have found them is trying to attack them and see their response: if they are protected, then as sure as hell they know about the vulnerability and how to exploit it too.

Re: This was a labour of love, so....

[Zero__Kelvin]

Not if you love security, fuzzing, and money :^)

Re:OK whew, authenticated users escalating/exploit

[AHuxley]

The threat model is gov/mil/security services.
https://en.wikipedia.org/wiki/... will find a VPN user.
"Showing the usage of virtual private networks (VPNs) and machines that can potentially be hacked via TAO."
Thats why most governments have few or no laws or comments about the about encrypted VPN use. Revealed: how US and UK spy agencies defeat internet privacy and security (September 2013)
https://www.theguardian.com/wo...
"... Edge hill's initial aim was to decode the encrypted traffic certified by three major (unnamed) internet companies and 30 types of Virtual Private Network (VPN) ..."