Study Finds Bug Bounty Programs Extremely Cost-Effective

itwbennett writes "U.C. Berkeley researchers have determined that crowdsourcing bug-finding is a far better investment than hiring employees to do the job. Here's the math: Over the last three years, Google has paid $580,000 and Mozilla has paid $570,000 for bugs found in their Chrome and Firefox browsers — and hundreds of vulnerabilities have been fixed. Compare that to the average annual cost of a single North American developer (about $100,000, plus 50% overhead), 'we see that the cost of either of these VRPs (vulnerability reward programs) is comparable to the cost of just one member of the browser security team,' the researchers wrote (PDF). And the crowdsourcing also uncovered more bugs than a single full-time developer could find."

Comments

Incentives

[Todd Knarr]

The major problem is that on-staff developers are usually discouraged from going on bug-hunts. Management would rather have them developing new features, so they won't allocate time towards finding bugs. When what the company policy towards finding bugs is conflicts with how your manager assigns you tasks, guess which one wins. Worse, most of the time an employee who ignores his to-do list to go find problems ends up penalized either explicitly (by bad reviews) or implicitly (negative impact from people being annoyed that he made work for them). Outsiders in these bounty programs don't have to worry about a manager assigning them 100% to new features and 0% to finding vulnerabilities and they don't have to worry about the impact of bad reviews or negative comments by managers about the extra work they created for everybody.

Re:Incentives

[VorpalRodent]

This.

And not just bug hunts. I have a laundry list of things that need to be refactored, but every time we think we might have a chance to do so, project management decides something else is more important. We have people complaining about things being slow, but when told that we need to spend time to make it faster, we instead get directed at new features or, worse, tweaks for the sake of a single non-representative customer that happens to have the ear of the project owner.

Re:Incentives

[CastrTroy]

Exactly. I think if you found the right kind of employee and told them to hunt for bugs all day long and get paid for it, They'd probably uncover quite a few bugs. Give them complete access to the code, source control, and test suites, and they could probably find bugs much more efficiently than getting somebody to find vulnerabilities from the outside.

dilbert

[Joe_Dragon]

http://dilbert.com/strips/comic/1995-11-13/

Re:dilbert

[CastrTroy]

I wonder if anything like this is going on internally. Let's say a developer at Google knows about a problem. He could either fix it, and get his regular pay, or he could tell his friend about the bug, and split the bounty with his friend who "discovered" the bug. Either way the bug gets fixed. And it probably get's fixed faster this way, since it's now an externally known vulnerability.

VRPs are the new sweatshops

[OleMoudi]

This is indeed true specially for popular companies with rather mature SecOps that pay minimum wages for vulnerabilities that are indeed hard to find or require a pretty darn good skill level to discover. Some of them even only offer swag in exchange of finding serious threats such as persistent XSS or authentication bypass. They maybe feature the researcher in some blog post to publicly thank him and attract the wannabe crowds.

Having said that, I myself have participated in several of these programs (with varying success) and come to realize that probably Google and Facebook are the only VRPs currently paying reasonable wages for bugs in terms of cost efficiency for the researcher.

On the other hand, some of us just enjoy from time to time trying to find security bugs for fun (maybe because we are huge nerds) so these programs offer a great opportunity to test things and not risking ending up in jail.

Cost of fiunding bugs != cost of fixing them.

[140Mandak262Jamuna]

Browsers have very large installed base. There are enough bug spotters even if a very small fraction of them actually hunt and report bugs. Even then, the bounty is for finding the bugs, not fixing the bugs that includes the cost of coming up with a fix, verifying it fixes the problem, testing to make sure it does not create new problems and rolling out the fix.

Ineffective, unfortunately

[gweihir]

This is effective for the low-hanging fruit, i.e. the easy (relatively) to find security-related bugs. For things that require advanced techniques or expensive tools (like Fortify), it fails. Unfortunately, the harder to find bugs are still well within reach of spy agencies of all kind, including a number that is allowed to do industrial espionage (like the US or France).

So while this looks good on the surface, it is really just making the problem worse. The only exception is software that has very low security needs.

For reliability, it is about as ineffective, as only easy to identify bugs will be tracked down.