Improving Signal Over 10,000 Bugs

On July 2nd, 2015, HackerOne reached 10,000 valid security vulnerabilities resolved on our platform – that’s 10,000 fewer ways for attackers to exploit users, steal data, and cause damage. The first bug was filed on the HackerOne platform 20 months ago, and today over 250 organizations resolve hundreds of bugs each week thanks to the help of thousands of researchers around the world.

Reflecting on the journey to 10,000 bugs, we thought it would be interesting to share some of the ways we’ve tackled the challenge of increasing signal on a fast growing platform. As a team, we spend a great deal of time improving automation and refining our platform algorithms to increase signal and enhance the customer experience.

Defining Signal

To help clarify our process, we can start by looking at how we categorize signal. At the highest level, we categorized vulnerability reports into three groups as response teams close them:

• Clear Signal – Vulnerability reports closed as “resolved”. This means the issue was a valid security bug that was fixed by the vulnerability response team.
• Nominal Signal – These reports are closed as marked “won’t- fix” or duplicates of resolved issues. While not contributing to clear signal, many of these reports were technically accurate based on the best information available to the researcher.
• Invalid – These reports are closed as “Not Applicable,” “Spam” or duplicates of these types. This represents the noise in the signal to noise ratio.

HackerOne Platform Signal

The chart below shows HackerOne’s clear signal compared to that of other programs sharing data publicly. From left to right, we see that the programs from GitHub, Facebook and Google document signal rates of 4%, 5%, and 7%, respectively. HackerOne’s baseline clear signal percentage was twice that of Google’s and roughly 3 times that of GitHub and Facebook’s.

Improving Signal with Reputation

After benchmarking the platform, we introduced Reputation, a system that scores researchers based on the validity of submitted reports. As expected, it had an immediate impact on improving signal. Three months after we introduced Reputation, we added basic Rate Limiting, which works by capping the number of reports that a low reputation researcher can submit in a given time frame. Our current algorithm is extremely lenient toward researchers who submit invalid reports. In other words, we can still make significant improvements to our signal ratio by further highlighting high-quality researchers and guiding new and low-signal researchers towards better behavior.

The chart above shows the progression of signal improvement we achieved by first adding Reputation and later Rate Limiting. The left-hand columns show a baseline of 14% clear and 33% nominal signal before adding Reputation. The middle columns show that adding Reputation raised clear signal to 20% and nominal signal to 41%. The right-hand columns show the impact of next adding Rate Limiting onto the platform, which improved clear signal to 23%, while nominal signal held steady at 41%.

The chart above shows how higher-reputation researchers drive the majority of clear signal on HackerOne. Inspecting the left-hand column, you’ll note that top 10% of researchers (top three sections in column) find 80% of valid vulnerabilities. By contrast, the right-hand column shows that the bottom half of researchers (lower section in column) account for nearly 60% of all invalid reports. With these new insights into our platform, we can better inform our future endeavors to refine the experience for both researchers and organizations.

Invitation-only Versus Public Program Signal

In contrast to limiting the number of invalid reports in our publicly-listed programs, our invitation-only programs draw participation only from our top reputation hackers, thus creating a high-signal environment for organizations. The chart below separates public and invitation-only programs on HackerOne, listing their respective signals. In public programs, the sum of nominal and clear signal is 53% compared to 85% in invitation-only programs.

10,000 Bugs and Beyond

HackerOne is committed to applying what we’ve learned from the first 10,000 bugs resolved to improving the platform for the next 10,000 bugs found and beyond. We look forward to helping organizations become more secure, creating opportunities for hackers, and making the Internet safer for all. From all of us at HackerOne, thank you 10,000 times.

HackerOne is the #1 hacker-powered security platform, helping organizations find and fix critical vulnerabilities before they can be criminally exploited. As the contemporary alternative to traditional penetration testing, our bug bounty program solutions encompass vulnerability assessment, crowdsourced testing and responsible disclosure management. Discover more about our security testing solutions or Contact Us today.