The Joys of Running a Bug Bounty Program

Trailrunner7 writes "When Barracuda Networks started its bug bounty program about three months ago, company officials weren't exactly sure what to expect. They didn't know whether there'd be an onslaught of submissions or the sound of crickets chirping. The reality turned out to be somewhere in the middle. Overall, the company has been getting about 10 bug reports a month, none of which has been very serious. But that doesn't mean the program hasn't been a success. Peck said that Barracuda also had run into the same problem that Google and others have: hackers don't pay much attention to directions. The company set out specific parameters for what kind of vulnerabilities in which products were in scope for the rewards, but some researchers still submitted flaws that were out of bounds, including bugs in partners' products or in the Barracuda corporate Web site."

Comments

Hackers and directions

[MrEricSir]

Hell, I could have told you that hackers don't read directions.

But would you have read my advice?

Re:Hackers and directions

[RavenChild]

Hackers are hackers because they don't follow directions. Using something as intended goes against hacking's very essence.

Re:Hackers and directions

[symbolset]

If you give web designers a place to submit bug reports on your website, even if it's not exactly topical, they'll use it. Some web designs are truly unfortunate. If HP published the physical location of their web design teams they'd probably have to enroll them in something similar to a witness protection program.

Re:Hackers and directions

[Anonymous Coward]

Wait, this was covered in Dilbert years ago.
The pointy-haired boss announced there would be bonuses based on bugs found.

Wally shouted "woo hoo, I'm writing myself a minivan today"

What?

[valkabo]

Hackers are excellent at following directions. They are just also excellent at seeing where the directions are flawed and exploiting them. What.. you think steve the hacker is finding holes in your software by guessing? No. He uses the program like it is suppose to be used and then tracks down the issues he is looking to exploit. You can't break a rule if you don't totally understand it.

Re:What?

[sortius_nod]

I'm thinking you don't either by saying that it's "guessing".

An informed guess is different to a blind guess, and to be quite frank, blind guesses don't generally find exploits or bugs.

That said, running in and guessing isn't what you do when you want to break a system, generally you need to know how the system works, or know enough to be able to theorise what might break it. It has nothing to do with guessing.

Re:What?

[totally bogus dude]

OT, but congrats... your post is currently scored at -1, Insightful.

A far cry from the coveted +5 Troll but still pretty cool!

Re:What?

[TimHunter]

You can't break a rule if you don't totally understand it.

Uh, what? Frequently you break a rule because you don't understand it.

Just for example, suppose I tried to upgrade the electrical wiring in my house without understanding the electrical system building code. The building inspector won't approve my changes because I broke the rules about electrical wiring.

Pay up if they fix the "out of bounds" issues

[PatPending]

The company set out specific parameters for what kind of vulnerabilities in which products were in scope for the rewards, but some researchers still submitted flaws that were out of bounds, including bugs in partners' products or in the Barracuda corporate Web site."

If they do in fact fix those "out of bounds" issues and/or its corporate web site then they should pay something to the discoverer. Only if they don't do anything about them should they not pay anything.

Re:Pay up if they fix the "out of bounds" issues

[Wrath0fb0b]

If they do in fact fix those "out of bounds" issues and/or its corporate web site then they should pay something to the discoverer. Only if they don't do anything about them should they not pay anything.

If I ask a contractor to assess my foundation and he tells me that my water heater is busted, do I owe him money if I later replace the heater? There was an explicit deal regarding which flaws qualify for bounties and which do not. If someone submits one contrary to an honest reading of those terms, they are owed nothing.

Re:Pay up if they fix the "out of bounds" issues

[scdeimos]

There are still "good citizens" out there that will report bugs without an expectation of payment.

One of our applications files cases for exceptions through FogBugz, giving users the opportunity to add their own comments before submission. We know some users just click Cancel (thus not reporting an issue) but maybe 10% of submissions have a comment and about 10% of those say anything meaningful to help us replicate the bug. I don't recall anybody asking for money before telling us what they did to break it.

Re:Pay up if they fix the "out of bounds" issues

[Voyager529]

That depends. You're right if you're asking him to limit his assessment to the foundation, however, if he sees that the water heater is set to burst in such a manner that when it does break that it will damage the foundation, then yes, I'd say you're still on the hook. At the end of the day, a risk to the foundation was found. If you're limiting the risk to only those which have already manifested, then yes the case could be argued, but you'd be a fool to not consider it an assessment within the scope of the question. There's a difference between that (finding a secondary answer to the question being asked) and simply saying that the heater is broken so your water won't be hot. I'd say that the former should still count, while the latter should not.

Re:Pay up if they fix the "out of bounds" issues

[noidentity]

The company set out specific parameters for what kind of vulnerabilities in which products were in scope for the rewards, but some researchers still submitted flaws that were out of bounds, including bugs in partners' products or in the Barracuda corporate Web site.

Who would complain that people are submitting more bug reports than asked for?!? They're getting reports for their website, without any need to pay a bounty. The problem with this is? Even bug reports of a competitor's product are useful in letting them know what areas are important to customers.

Re:Pay up if they fix the "out of bounds" issues

[kaiser423]

If it saved you money and/or fixed a problem; aka his service reaped unexpected obunties, I would think that the respectful thing to do would be to pay him. But then again, I do actually want people to tell me when other things outside of scope are wrong, because, you know, that's helpful and worth something.

Re:Pay up if they fix the "out of bounds" issues

[hedwards]

That's partially because we're used to bug reports that go somewhere, and we have absolutely no clue as to what exactly is done with the information. And often times we're not told what information is being sent anyways. I don't know what things are like where you're working, but I do know that a lot of people aren't going to trust random strangers. Which in a sense is odd, given that there's enough trust to run the program, but there you go.

Re:Pay up if they fix the "out of bounds" issues

[JWSmythe]

    Actually, it's much different than that.

    It would be like you hired a contractor to assess the foundation of your house (your application), and instead he tells you about problems the front door on the adjoining house (your website), or about the foundation of houses in another state (competitors applications). Only an idiot would pay for such a report.

    If (and only if) they asked for a comprehensive evaluation of the security of their company, would the web site be included in it, unless the web site is essential to the operation of their application. If they intended to get paid for the work, they shouldn't have reported it through the bug tracking system. It could have been reported independently. I didn't read far enough to see if the bugs were really security bugs, or if they were simply rendering errors.

Re:Pay up if they fix the "out of bounds" issues

[SharpFang]

Actually, they are owed gratitude and what little courtesy demands. You have no contractual obligation to reward them, but in all fairness, if they discovered an error you didn't know about, where you didn't expect it, they deserve some kind of gratitude.

Re:Pay up if they fix the "out of bounds" issues

[TubeSteak]

Water heaters aside, I think you'd be wise not to piss of people who have shown they can find holes in your product &/or corporate website, regardless of their ability to follow directions.

Re:Pay up if they fix the "out of bounds" issues

[scdeimos]

Yes it is rather strange, isn't it (trusting you enough to run your application but not enough to log error reports)?

We go to a great detail of trouble showing the user what information will be logged (given that this is running inside the application's exception handler we have to be very careful about triggering more exceptions) so that users can make an informed decision about the Ok/Cancel buttons on the exception dialog. Still, we occasionally get users (staff or customers) that complain to us through other channels (phone/email) about problems that they've been having and then have to fess-up that they've never bothered to log an exception report.

Sometimes they've been experiencing the problem for months and we could have fixed it in a couple of minutes and had it out in the wild a couple of releases ago. Maybe we should take a leaf out of Sony/BMG's book and just do it without their consent. (jk)

Re:Pay up if they fix the "out of bounds" issues

[blackest_k]

It's generally really difficult to get through to anyone who could actually get a bug fixed.

the sales contacts won't be able to do anything
dave from bangalore can't deviate from his scripts and will not pass anything back to the engineers on the other side of the world. so who , where?

so given the opportunity to pass bug reports back to a company, why not pass your bug information to them. it's not like there is an alternative point of contact available for these other bugs.

What is really crazy is that the corporate mind thinks this is people wanting to get paid for fixing unrelated bugs it isn't. It's people wanting bugs fixed even if it is a bug in the partners software at least corporation a has a line of communication to corporation b and hopefully the bug can be resolved to the benefit of both corporations and their customers.

At least with open source software you can submit a bug about anything and hope to see it get fixed or find work arounds or updates or if needs be fix it yourself (it is always going to be easier for someone who has worked on the code base to find the error fix it and submit a patch).

Re:Pay up if they fix the "out of bounds" issues

[bWareiWare.co.uk]

The question about what is under the contract is only part of the issue. If an unsolicited observation saves you money (i.e. you wouldn't have noticed yourself and you were able to take preventative action) it would make sound economic sense to express your gratitude.

Re:Pay up if they fix the "out of bounds" issues

[Abstrackt]

The question about what is under the contract is only part of the issue. If an unsolicited observation saves you money (i.e. you wouldn't have noticed yourself and you were able to take preventative action) it would make sound economic sense to express your gratitude.

Interesting point. One thing isn't clear to me though, and I'm honestly curious, what constitutes an acceptable gesture of gratitude? Is a company required to express their gratitude towards someone's observation with money? Or is it enough to give them a personalized "thank you" in an email, offer them a free copy of future versions of the software or simply give them some public recognition?

Barracuda

[American AC in Paris]

...does "your messaging client is such a kludge that I would frankly rather try use an actual elongated carnivorous fish to IM with my co-workers" count as a bug?

Re:Barracuda

[Jerf]

I was the team lead on that product for a long time. It's based on a standard XMPP server. Use any standard XMPP client you like, if your administrator lets you install it. (If not, well... I can't really solve that problem.) The shipped client has deliberately been simplified for non-power users, as a result of a lot of feedback from such people. For example, XMPP's resource handling confuses most people, so it has been hard-coded in the client. If you're a power user you should definitely use Pidgin or Trillian or something.

Puny bounties

[Animats]

There was once a real-time OS company that gave you a Bug, a Volkswagen Beetle, if you found a bug in their OS. They gave away about two cars a year, and it was worth it.

Re:Puny bounties

[hedwards]

That's always a problem, there's a balance, you can't give away something that expensive for every bug no matter how tiny. But you also do have to compensate researchers enough to make it worth their while for more important bugs.

Re:Puny bounties

[syousef]

That's always a problem, there's a balance, you can't give away something that expensive for every bug no matter how tiny. But you also do have to compensate researchers enough to make it worth their while for more important bugs.

So only guarantee small bounties, but then generously offer disgressionary rewards for important bugs then use the publicity when you do give away something of value to get more researchers interested. Still cheaper than highering a traditional test team.

Re:Puny bounties

[im_thatoneguy]

There was once a real-time OS company that gave you a Bug, a Volkswagen Beetle, if you found a bug in their OS.

The latest generation of VW Bug? Why not just knock them over the head and steal their children? Why on earth would anyone submit a bug if there was a chance they might have to drive a VW Beetle?

Not Just Hackers

[Bieeanda]

I hate to break it this way, but most people don't have the QA skills of a goldfish. Most of them, even given guidelines, walkthroughs, or even formal instruction on how to write a bug report, would rather just drop a single, unhelpful line and get back to waiting for a cheque.

Re:Huh. Really?

[hedwards]

Let me guess, you work at Microsoft, don't you?

Heh . .

[228e2]

I remembered finding a bug in their bug submission portal . . . it was right of me to never submit it, right?

Re:Heh . .

[SharpFang]

Depends what it was concerning. Same problem as running fsck from a corrupted filesystem: you have no warranty fsck itself is not corrupted and won't corrupt the filesystem further.

Legal vs ethical.

[khasim]

If I ask a contractor to assess my foundation and he tells me that my water heater is busted, do I owe him money if I later replace the heater?

Legally, that would depend upon the specific wording of the contract you signed with him.

Ethically, if he found a flaw that you did not know about and told you about it in a manner that allowed you to save money by fixing it before it got worse, then yes, you do owe him.

I'm working on a better bounty program

[jimktrains]

I was in the middle of writing a site for bug/feature bounties that any project could sign up and use, but I'm not quite able to demo it yet. I've slowed work because I got tons of negative feedback on the idea from people thinking that it's a beaten concept and there was no reason to write a (better) app since others are out there. I'm still working, but slowly.