Measuring Success in Vulnerability Disclosure

By HackerOne Customer Success and Data Science Teams

At HackerOne, we’re deeply interested in the success of vulnerability disclosure programs, and are constantly striving to better understand just what drives their success. To shed light on what contributes to a successful program, we’ve been analyzing our unique set of data from hundreds of organizations.* Based on this, we’re excited to share the HackerOne Success Index (HSI), a method to measure the effectiveness of HackerOne-powered vulnerability disclosure programs. The index calculates six dimensions, from 1 to 10, by which programs can benchmark their success each month. We briefly discuss each dimension below, and we’ll explore them in more depth over the course of this series.

Success is Multidimensional

Our investigation shows that success doesn’t simply come from doing well on a single dimension, but rather across a combination of them. Successful HackerOne programs — those that consistently receive valid, security-enhancing reports — excel in a variety of the six distinct but interconnected dimensions below:

1. Vulnerabilities Fixed

Simply put, to be a thriving program, you need to receive and resolve vulnerability reports. The most successful programs also receive a wide array of vulnerability types across different security aspects. Performance in the other indexes will affect the volume and quality of vulnerabilities fixed.

2. Reward Competitiveness

Higher bounties tend to attract higher reputation researchers who find more severe vulnerabilities, though there isn’t a simple linear relationship between reward level and activity. In fact, as our index quantifies, there are successful programs that offer no financial rewards at all.

3. Response Efficiency

Researchers appreciate clear, timely communication. The data show that programs that respond quickly to new reports, and keep open communication channels during the triage and resolution process, tend to get more reports and more repeat researchers, leading to a virtuous, security-enhancing cycle. In addition, the timely resolution of vulnerabilities reduces the risk of potential exploitation, leading to greater security.

4. Researcher Depth

Researchers who repeatedly investigate your products are going to find more severe vulnerabilities as they learn your code. It’s (data) science. Not to mention that repeat researchers tend to produce better reports, and have smoother communication with your team, as you work together over time. This metric also takes into account the Reputation of contributors, since the data show that high reputation researchers are more capable of finding critical issues.

5. Researcher Breadth

This is where Linus’s law, “given enough eyeballs, all bugs are shallow,” really kicks in. With a large-enough testing group, problems in your code will be found quickly and fixes identified more efficiently. This is one of the reasons successful HackerOne programs continually add new researchers until ultimately opening up publicly, at which time they leverage the greatest potential testing pool on the planet: the entire population of the Internet.

6. Signal Ratio

The measure of valid reports against the total number of issues received is a primary indicator of the value gained from a program (check out our blogpost on signal). A high signal ratio means more actual vulnerabilities identified, and ultimately fixed, for the same amount of time spent triaging and responding. While we’ve made great strides in improving signal across the platform, it remains our top area of focus, and we have additional enhancements coming soon.

The result of putting these dimensions together is an advanced framework for quantifying impact and assessing the performance of these programs.

*Factors are ordered by their weights

Successful programs neither display a single HSI profile, nor necessarily have high marks in every single dimension. These indices will reflect a variety of circumstances, notably the program’s goals and organizational characteristics like security maturity, size, and attack surface.

Take, for example, these spider chart visualizations of the HSI for two successful programs, graphically representing two large enterprise programs: one that offers bounties, and one that does not. Program 1, on the left, is one of the most successful programs in our dataset, topping the charts for Vulnerabilities Fixed and Researcher Breadth and Depth–advantages for public programs–and getting high marks in Reward Competitiveness as well. Program 2, on the right, also does very well in most dimensions, despite offering no monetary bounty at all. These examples suggest two things. First, you can clearly have a successful disclosure program without offering bounties, but with a slight cost to Researcher Breadth and Depth. Second, you should ignore dogma and use data to determine which incentives produce the ideal outcome for your organization and its unique circumstances.

Over the coming weeks, we’ll further explore these dimensions in a series of blog posts that describe what goes into each one, show data on why that facet of the program is important, and make recommendations for how programs can improve their performance. As we operationalize the HSI, we are exploring ways to make it accessible to all HackerOne programs on an on-going basis. Contact the team now for a preview of your program’s Success Index.

**Note: The Success Index is based entirely off of transaction data with no access to teams’ vulnerability information.*

HackerOne is the #1 hacker-powered security platform, helping organizations find and fix critical vulnerabilities before they can be criminally exploited. As the contemporary alternative to traditional penetration testing, our bug bounty program solutions encompass vulnerability assessment, crowdsourced testing and responsible disclosure management. Discover more about our security testing solutions or Contact Us today.