Slashdot Mirror
Study Finds Bug Bounty Programs Extremely Cost-Effective
itwbennett writes "U.C. Berkeley researchers have determined that crowdsourcing bug-finding is a far better investment than hiring employees to do the job. Here's the math: Over the last three years, Google has paid $580,000 and Mozilla has paid $570,000 for bugs found in their Chrome and Firefox browsers — and hundreds of vulnerabilities have been fixed. Compare that to the average annual cost of a single North American developer (about $100,000, plus 50% overhead), 'we see that the cost of either of these VRPs (vulnerability reward programs) is comparable to the cost of just one member of the browser security team,' the researchers wrote (PDF). And the crowdsourcing also uncovered more bugs than a single full-time developer could find."
Ohhh, code review is useful???
The major problem is that on-staff developers are usually discouraged from going on bug-hunts. Management would rather have them developing new features, so they won't allocate time towards finding bugs. When what the company policy towards finding bugs is conflicts with how your manager assigns you tasks, guess which one wins. Worse, most of the time an employee who ignores his to-do list to go find problems ends up penalized either explicitly (by bad reviews) or implicitly (negative impact from people being annoyed that he made work for them). Outsiders in these bounty programs don't have to worry about a manager assigning them 100% to new features and 0% to finding vulnerabilities and they don't have to worry about the impact of bad reviews or negative comments by managers about the extra work they created for everybody.
Mostly shows how being good at finding bugs is a different skill than being good at job interviews.
and no one is there to hear it does it make a sound ??
NO !!
Bounties are like telling hit-men to take you out but then paying them if they tell you about it first !! Bounties are Yahoo-think !!
The bounty system is the ideal tool to exploit thousands of people working for free, so that only one has to be paid. The best thing is that you only have to pay for results.
isn't $570,000 / $150,000 about 3.8 people? (articles numbers.) Still probably a good deal, but not quite as good.
...in their methodology, but first, pony up.
http://dilbert.com/strips/comic/1995-11-13/
This is indeed true specially for popular companies with rather mature SecOps that pay minimum wages for vulnerabilities that are indeed hard to find or require a pretty darn good skill level to discover. Some of them even only offer swag in exchange of finding serious threats such as persistent XSS or authentication bypass. They maybe feature the researcher in some blog post to publicly thank him and attract the wannabe crowds.
Having said that, I myself have participated in several of these programs (with varying success) and come to realize that probably Google and Facebook are the only VRPs currently paying reasonable wages for bugs in terms of cost efficiency for the researcher.
On the other hand, some of us just enjoy from time to time trying to find security bugs for fun (maybe because we are huge nerds) so these programs offer a great opportunity to test things and not risking ending up in jail.
---------
Thinking never hurt anybody --MacGyver
as a corporation is abdicates you from the responsibility of things like health insurance in countries like america that have very expensive coverage individuals typically cannot afford. In more advanced countries like sweden or canada, youre indirectly allowing a government to subsidize a component of your under-the-table employment of coders and hackers. expenses like retirement, life insurance, dental coverage and the cost of work-related activities like ice cream socials are then realized as a savings. In my opinion coders and hackers must be very careful when engaging in bug bounty as the cost of a programmer including benefits is often not fully reimbursed when they find and patch a bug. even if that is not a primary consideration, the ethics of fixing googles problems are worth considering
small projects like mozilla should get to do it, as theyve consistently demonstrated a moral and ethical commitment to protecting the internet for all humankind. Google, a major multinational corporation that lobbies congress for H1B legislation, is in a bit more of a grey area. Chrome is an offering in which its user becomes the product, the final objective to sell the subjects data to various other corporations and earn a profit.
Good people go to bed earlier.
Maybe they should have compared the salary of a QA person instead of a developer. As a developer, I find lots of bugs, and then fix them. I also fix the bugs that QA finds, but usually spend a lot of time trying to figure out how to reproduce the issue ("uhh, first I clicked on this and then I clicked on that and then something weird happened").
Anywhile, it's hard to crowd source a product that has not been released yet and most companies don't have the fan-bois and gurls to even consider this strategy.
When the PHB announced a bug bounty program, Wally vowed to write himself a new car that afternoon.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Browsers have very large installed base. There are enough bug spotters even if a very small fraction of them actually hunt and report bugs. Even then, the bounty is for finding the bugs, not fixing the bugs that includes the cost of coming up with a fix, verifying it fixes the problem, testing to make sure it does not create new problems and rolling out the fix.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
So if they cut the developer & the tester (bugs get found/fixed in teams), they get a lot more bugs released into the wild, a lot more bad PR, and lot more bad user experience.... oh, but it's fine, they're saving a few thousand bucks a year..
That means that there is a strong incentive for companies to create insecure, crappy software and then let so-called "white hat hackers" fix their bugs at a discount. And because any other form of disclosure is illegal, the companies are pretty well protected from negative consequences of their bugs and deflect from their own negligence by blaming "black hat hackers".
Given that disclosure is also at the terms of the payer, you also get less transparency versus independent disclosure.
Twitter supports and protects racists - by smearing their critics with the "Hate Speech" label.
This study may neglect to mention the benefits of advertising and exposure even if only ten people found the bugs.
Because the sort of programmer that's good at finding/fixing these bugs...is not the sort of programmer that the interview process determines would be a "good fit" for the organization.
"Once we've identified and embraced our sickness, we'll have strength...and that's when we get dangerous." - John Waters
This is effective for the low-hanging fruit, i.e. the easy (relatively) to find security-related bugs. For things that require advanced techniques or expensive tools (like Fortify), it fails. Unfortunately, the harder to find bugs are still well within reach of spy agencies of all kind, including a number that is allowed to do industrial espionage (like the US or France).
So while this looks good on the surface, it is really just making the problem worse. The only exception is software that has very low security needs.
For reliability, it is about as ineffective, as only easy to identify bugs will be tracked down.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I would hope Google is smart enough to know that you don't need an experienced developer to find bugs in their code. Aspiring developers fresh out of college are more then adequate. At 50k a pop google could have hired 7 PFYs spending 14,000 hours scouring code, hell give them 1k bonuses for each bug to keep them motivated.
Knowledge = Power
P= W/t
t=Money
Money = Work/Knowledge so the less you know the more you make
I wonder if anything like this is going on internally. Let's say a developer at Google knows about a problem. He could either fix it, and get his regular pay, or he could tell his friend about the bug, and split the bounty with his friend who "discovered" the bug. Either way the bug gets fixed. And it probably get's fixed faster this way, since it's now an externally known vulnerability.
Let's put it this way, consider ALL the knowledge, experience, and hours and hours of trying to figure out these bugs.
I wonder what the actual hourly benefit is to these bug catchers.
Composer analogy: A music prof in college I had described a piece he was commissioned to write. He said he was paid $2,500 ('Ooos' from class); which based on the number of hours he put into it amounted to roughly $2.50 per hour.
Not so good.
I see some of these bounties and all I can say is that these folks ain't doing it for the money.
On the other hand, the notoriety of finding a bug with GOOGLE, I'm sure, is a GREAT resume builder!
What I'm really shocked about is that you need a university to figure this out. Or rather do research on this. Companies figured this out quite some time ago and anyone with a functioning brain can see why. :)
What I'm more interested in is that king of people spend their time in participating in programs like this. The chances that you find a bug are not that big. The financial reward, given the amount of time you will spend on finding a bug is probably also relatively small.
From a company's point of view on the other hand, it's great. Many people working for you. For free. A job well done
Privacy is terrorism.
or one instead offers ``certificates of deposit'' in the (fictional) ``Bank of San Seriffe'': http://en.wikipedia.org/wiki/Knuth_reward_check
William
(who is quite bummed that he didn't get his reward check back when Dr. Knuth was using Wells Fargo as his bank: http://www.truetex.com/knuthchk.htm )
Sphinx of black quartz, judge my vow.
These amounts of money were paid for the bugs that bounty hunters bothered to report.
The question is, how many bugs still exist in the software, and of those, how many have been discovered and not reported?
All this article is doing is confirming that, in this case, if you make an effort to minimize spending, your costs can be reduced.
It does not say what kind of value you are receiving for what you pay.
Is good to reward people that find security holes, at the very least because is a safer bet than selling them in the black market, or keeping them for yourself or the government to exploit them. But it should not be a replacement for actually having dedicated people activelly working for your security that will report to you if something weird is there, some could actually go to the black market (or be found by government teams and never disclosed that it is there because is an useful cyberweapon) and you must be proactive from your side
Developers are not paid to be dedicated bug hunters. They deliver new code and maintain the code base. They should be trained in security, but the expectation of flawless perfection in secure coding is absurd.
At many software companies, a dedicated person or team performs the audit and pen testing function. The number of issues the team finds are exponentially more than are reported. And many are never acknowledged, but are fixed in the latest releases. Stay up to date, really.
The article gives examples of companies that have just a few products. Consider the budget of running a bug bounty program for a company that has 50-300 products - it would easily run into the millions.
Bounty programs are great if the money is there (I'm looking at your billions in profit Microsoft!) But also think about companies who barely profit and have greater needs for survival.
Every piece of software you use likely is vulnerable to something that is not yet discovered, fixed, or publicly disclosed. It's about what the company is doing to reduce those vulnerabilities and how well they handle the inevitable vulnerability report.
It sure isn't the average in Canada.
It's not surprising at all that piecemeal work, with no provision for healthcare, vacation etc. - much less reliable, ongoing income - is more profitable for business.
Why should technology workers be intrigued or inspired by this? Why is this information presented to technology workers as another avenue to praise Google's or Mozilla's cleverness? And why do technology workers so consistently dig their own graves by latching onto this kind of ideology and failing to fight for labor rights?
where you have millions of folks looking at your free software for long periods of time. If you're a commercial software vendor, however, with a $10,000 non web-based package and at most a few thousand users (There are still a *lot* of these), then this approach is very unlikely to succeed. Commercial software users are rarely interested enough to report a bug that doesn't actively interfere with their daily work.
Please do not read this sig. Thank you.
What the hell does QA do these days anyway?
RIght, with shaming. That's a good idea:
"Implement un-specced feature X in unreasonable time Y on top of an unstable foundation and no you can't take the time to refactor it we essentially want you to produce lava-code, and no you can't have time to test it, we promised it to the customer yesterday and no we don't do QA around here: 'just do it', 'make it so' and get it into production.
Oh. Oh look. There's a software defect!
Well you must be a very incompetent developer then 'innit mate. Perhaps if I engage in "shaming behaviour", express my "dissapointment" at your inability to defy logic and the laws of physics, threaten you with penalties and complain a lot for you to "go faster" I can hide the fact that I "manage" software projects for a living without, you know, actually knowing what software is. Oh and by the way give me a cast-iron reliable estimate on the following utterly vague and large unknown feature set right now, just pull it out of your ass and i'll hold you to it and no you can't have any time for planning."
Right dude, seriously: I guess you never worked as a developer then. You'ld also make a terrible manager.
Addendum:
"Right OK guys, meeting with the client went well, they've hired us for another project and it's a big one so we really need you to perform for the team on this... Guys? Guys? Hey, where'd everyone go?"
Considering that they've both been caught stiffing the crowd when crowd-sourcing, it's little wonder that they're saving money with this approach: you stiff an employee, and you'll get sued.
... software developers contribute to their own devaluation. Imagine plumbers or electricians getting together in their spare time to repair or upgrade a corporate building, in the vague hope that the corporation in question might give them a few beers, or a cap with a logo. This is why the management class continues to see IT / development as a bunch of easily satiated morons, who are more than willing to work hard for free pizza and swivel chairs.
The longer I'm in this field, the more convinced I am that it is not a long-term career option for grown-ups. Become a consultant (and hope no one reads past the weasel-words) or choose a technical career with a focus on physical deliverables.
Ha! Because the real money is in selling these bugs to foreign governments!
Cost effective!? Yeah, I'll fix your bugs for $1 million, since that's what I can make
selling them to France.
So what this encourages shops to do is NOT find/fix bugs (that could ruin their credibility) and instead simply release the software to the general public (lurking zero-day exploits et al) for the masses to find and suggest bug fixes. All the while, innocent consumers of the software have little guarantee that a good-faith effort was made on the part of the software-maker that they are PURCHASING software that will perform as advertised without exposing the user to undue risk.
Tell me companies won't be stupid enough to overlook this fact?
Major companies should have a contest, because then they'll get free work from everyone who enters and only have to give out one award. The current way they're doing it could be made more efficient.
While this bounty practice is now more or less restricted to software bugs, I sure hope it will never extend it's reach to other areas of software development and that it'll prove to be a vain and passing fad. Sure, the high bounties are appealing and so is the prestige that comes with them but this sort of "winner takes all" approach is tremendously wasteful regarding the work values of discarded competitors - not to mention, it's against the very idea of collaboration and one of the best way to polarize the software development community. Of course, it's no surprise that this activity is highly profitable to the lottery owners (is there really such a thing as a bankrupt casino?). In the end, I really hope those bounty programs will be regarded only with contempt and disdain by developers for it's really a farce towards our profession. (And for those wondering: no I'm not bitter about these bounty programs because I've wasted my time on the; I make a point not to participate in it).
Testers do most of the testing, and they're cheaper. Also, you get ehat you pay for, naive testers file stupid baive bugs, not the ones that really ferret out the big problems.
I get paid to audit code, so I'm biased.
The article says that no one employee could find hundreds of bugs and that's true. But when you hire employees you are building a process. Improving the process by writing a new QC script can eliminate hundreds of bugs over a couple years. These are not attributed to one employee and since the offending code is not committed then they aren't even counted as bug fixes.
Offering a bug bounty, on the other hand, is a unpredictable thing and you'll get random fixes. It is valuable because it provides a fresh perspective.
My guess is that if you collect a few bug bounties then Google will send you a recruiting email. It might be more expensive to hire you to work full time it's still a worthwhile thing.
developers don't "create" bugs, we don't sit down and say, hey, lets create a bug! and then go about making one, most of the time, we believe our code doesn't have bugs, cause if we thought it had bugs, we'd write it a different way, or we'd know about the bug in the first place and it'd be in our list of things to fix, normally those things are fixed quickly because we knew it was there, but things we don't know about, well, how do you expect us to find it? I didn't find it whilst I was writing the code and I'm the brain doing the typing, if I can't find it then, what do you think the chance of me finding out afterwards will be?
other people are very good at finding bugs in my code, if they exist, because they have a different mindset and think about things differently than I do, they think of a circumstance I didn't think of, great, you found the bug! but I didnt know it was there.....
so I think it's kind of normal that other people will find bugs in your code that you didn't know existed, so it makes perfect sense to reward those people to find them, paying ME to find them is going to not pay off in a big way, cause if I knew how to find bugs in my own code, I would have done it already and fixed it.
So UC Berkeley should compare the number of bugs found by researchers vs the number that Google's Internal QA Dept has found.
Then we'll know if it was really worth while. Since Google would never publish the number of bugs they find internally, all this data is worthless.
It is nice though getting people to QA your projects for basically free.
on the type of bug. If it's security related, you can never be sure that the customer that finds it will report it instead of exploit it.
In the world where the Microsofts(among) others demand an NDA to go along with the bug, many reported bugs stay unfixed.
I have some experience with (very closed source) software companies and bugs. They had a bugs list that users would report. They would rank customers by seat licenses, and give customers votes based on the number of seat licenses they had to vote on which bugs should be fixed. The company had an idea of how hard it would be to fix each problem, and if a bug didn't have the right threshold of votes, it wouldn't be fixed.
As the saying goes, an extra pair of eyes goes a long way...having numerous pairs goes even further, up until a certain point of diminishing returns and lower signal to noise ratio. Even so, I believe the real value of these programs is the fact that they're tapping into endless combinations and permutations of testing environments that in-house developers do not have access to. I've heard this approach labeled 'in the wild' testing that differs from 'in house or in the lab' approaches. Interesting and apparently quite effective.