Are bug bounty programs really working?
As Yelp launches a public bug bounty program we take a look at some success metrics, and how they apply to wider enterprises.
Yelp has announced that it has launched a public bug bounty program with HackerOne, offering a maximum reward of up to $15,000 USD for the most impactful exploits.
On its dedicated HackerOne page, Yelp threw down the gauntlet to hackers, but with some sensible caveats: “We want you to bring out your big guns, but hold off on actually breaking anything. Please avoid DDoS’ing us or breaking our systems and services while you are testing”, the consumer listings business said.
The HackerOne page shows updates on payouts, and a quick look reveals that in less than 24 hours two hackers have already picked up $100 each for their work. The company has also put together a ‘bug bounty map’, which it hopes will guide white-hat hackers into hunting the most useful bugs.
The move comes after two years of Yelp running a private bug-bounty program, which resulted in “fixing over a hundred potential vulnerabilities, and have paid bug bounties to dozens of security experts.” So how effective are bug hunting programs in 2016, and what are the wider pros and cons?
Also known as Vulnerability Reward Programs (VRPs), bug hunting programs vary widely in their implementation, but broadly seek to harness the freelance skills of hackers in return for hard cash. Their history has been somewhat checkered, not only due to their corporate sponsors not taking them seriously (see Yahoo’s t-shirt gate, where submitting serious vulnerabilities was rewarded with a free t-shirt
San Francisco-based HackerOne claims to be the first vulnerability coordination and bug bounty platform, and was created by security leaders from Facebook, Microsoft and Google, according to the company. It claims that in 2016 600 hackers participated in the HackerOne bounty program, submitting approximately 1,500 reports. This translated to resolving 58 valid security vulnerabilities, and rewarding 41 unique hackers a total $41,100, with an average payout of $1,082. Participation by hackers - not success note, but more of an engagement metric - was up 20 per cent in 2015 over 2014.
Meanwhile, in a different, but parallel world, Facebook reported that their bug hunting team classified 102 bug bounty submissions as high impact in 2015, an increase of 38 percent over 2014. Facebook received 13,233 total submissions in 2015 alone, from 5,543 researchers in 127 countries, and paid out $936K to 210 researchers, who submitted a total of 526 valid reports. That’s an interesting overall stat that Facebook didn’t shout about, a success rate of just under 4 per cent (3.97 per cent) in submitted reports, or an admission that 96% of the submissions were junk - or incomplete. However, the same maths goes for the HackerOne figures - a hit rate of just under 4 per cent (3.86 per cent).
Of course, having an open submissions system for a consumer facing brand such as Facebook is always going to generate a considerable pile of random correspondence, which may be why Apple is so late to the table, only recently announcing an invite-only program to launch September 2016. The rewards will range between $200,000 and less than $25,000. In an unusual move, Apple will encourage people who receive rewards to donate them to charity, and the Cupertino company will match donations to approved institutions.
Setting up a bug bounty scheme might be good business for tech giants, but might not be the same for your smaller enterprise however. There are many pitfalls to consider, as the figures above show, real financial commitment to process the submissions is required, and timeliness is key. When serious vulnerabilities are discovered then acknowledging them - and patching them - is key to maintaining good relations with the Infosec community. Bad feeling over unpaid bounties, or slow responses can easily snowball into serious brand damage, and it’s also worth remembering that even if you restrict the type of flaws you’ll pay out on (as Yelp has tried to do with their bug map), you may be surprised to find how many serious flaws will be found in those areas, pushing up costs. For a full rundown of the risks, SC magazine has a good piece here.
Overall it’s fair to say that given enough processing manpower, and a bottom line that can sustain a 4 per cent success rate, bug bounty programs are here to stay. However, imagining that they’re applicable to smaller businesses or a replacement for more structured penetration testing is a very long shot indeed.